Utils Module¶

This module provides a set of functions that may be used in multiple modules. The scope of this module is to provide functions that are not specific to any particular module, to decouple the modules from each other and to avoid code duplication.

splatnet3_scraper.utils.delinearize_json(keys, values)¶

Delinearizes a JSON object. Given a list of keys and a list of values, this function will return a JSON object. Turns a table into a JSON object.

Given a list of keys and a list of values, this function will return a JSON object. The keys are expected to be in the format of “key1.key2;index1.key3” where the semicolon indicates a list and the period indicates a nested object. The values are expected to be in the same order as the keys.

Parameters:¶

keys : list[str]¶: The keys of the JSON object. The keys are expected to be in the format of “key1.key2;index1.key3” where the semicolon indicates a list and the period indicates a nested object.
values : list[Any]¶: The values of the JSON object. The values are expected to be in the same order as the keys.

Returns:¶

dict[str, Any] – The JSON object created from the keys and values.

Return type:¶

dict[str, Any]

splatnet3_scraper.utils.enumerate_all_paths(data)¶

Recursively enumerates all paths in the given data.

Given a dictionary or list, returns a list of all paths in the data. For example, given the following data:

>>> data = {
...     "a": 1,
...     "b": {
...         "c": 2,
...         "d": [3, 4],
...     },
... }

The following paths would be returned:

>>> enumerate_all_paths(data)
... [
...     ("a",),
...     ("b",),
...     ("b", "c"),
...     ("b", "d"),
...     ("b", "d", 0),
...     ("b", "d", 1),
... ]

If the given data is not a dictionary or list, an empty list is returned.

Parameters:¶

data : dict | list | Any¶: The data to enumerate. If the data is not a dictionary or list, an empty list is returned. Otherwise, the function recurses into the data.

Returns:¶

list[tuple[str | int, …]] – A list of all paths in the given data.

Return type:¶

list[tuple[str | int, …]]

splatnet3_scraper.utils.get_fallback_hash_data()¶

Gets the fallback hash data for the GraphQL queries.

Loads the fallback hash data from the splatnet3_webview_data.json file and parses it to get the hashes for the queries.

Returns:¶: tuple[dict, str] – dict: The hash map for the GraphQL queries. str: The version of the hash map.
Return type:¶: tuple[dict, str]

splatnet3_scraper.utils.get_hash_data(url=None, ttl_hash=None)¶

Gets the hash data for the GraphQL queries with a time-to-live (TTL) cache.

Parameters:¶

url : str | None¶: The URL to get the hash data from. If None, the default URL will be used, from imink. This can be found in the GRAPH_QL_REFERENCE_URL variable. Defaults to None.
ttl_hash : int | None¶: The hash to use for the TTL cache. This is used to determine if the cache should be used or not. The hash is calculated by dividing the current time by the expiry time in seconds and rounding to the nearest integer. If None, the default expiry time of 15 minutes will be used. Note that this method of TTL caching does not guarantee that the cache will be invalidated after the expiry time, but it is a good enough approximation. Defaults to None.

Returns:¶

dict[str, str] – The hash map for the GraphQL queries.
str – The version of the hash map.

Return type:¶

tuple[dict, str]

splatnet3_scraper.utils.get_splatnet_hashes(url=None)¶

Gets the hashes for the GraphQL queries.

Uses requests to get the imink GraphQL query hash map JSON file and parses it to get the hashes for the queries. The initial request response contains two keys: hash_map and version. The hash_map key is the dictionary that contains the hashes for the queries and is what is returned by this method. The version key is the version of the hashes and is used to check if the hashes are up to date, it is used elsewhere in this package, but is not used here.

Parameters:¶

url : str | None¶: The URL to get the hash data from. If None, the default URL will be used, from imink. This can be found in the GRAPH_QL_REFERENCE_URL constant.

Returns:¶

dict[str, str] – The hashes for the GraphQL queries. The keys are the names of the queries and the values are the most up to date hashes for the queries.

Return type:¶

dict[str, str]

# noqa: DAR401 ValueError

splatnet3_scraper.utils.get_splatnet_version(url=None)¶

Gets the version of the GraphQL queries.

Uses requests to get the imink GraphQL query hash map JSON file and parses it to get the version of the queries. The initial request response contains two keys: hash_map and version. The version key is the version of the hashes and is what is returned by this method. The hash_map key is the dictionary that contains the hashes for the queries and is used elsewhere in this package, but is not used here.

Parameters:¶

url : str | None¶: The URL to get the hash data from. If None, the default URL will be used, from imink. This can be found in the GRAPH_QL_REFERENCE_URL constant.

Returns:¶

str – The version of the GraphQL queries.

Return type:¶

str

# noqa: DAR401 ValueError

splatnet3_scraper.utils.linearize_json(json_data)¶

Linearizes a JSON object. Given a JSON object, this function will return a tuple of keys and values. Turns a JSON object into a table.

Given a JSON object, this function will return a tuple of keys and values. The keys will be a tuple of strings, and the values will be a list of values. The keys will be the path to the value in the JSON object, and the values will be the values of the JSON object. For example, given the following JSON object:

>>> json_data = {
...     "a": 1,
...     "b": {
...         "c": 2,
...         "d": 3,
...     },
...     "e": [4, 5],
... }

The function will return the following tuple:

>>> linearize_json(json_data)
... (
...     ("a", "b.c", "b.d", "e;0", "e;1"),
...     [1, 2, 3, 4, 5],
... )

Parameters:¶

json_data : dict[str, Any]¶: The JSON object to linearize.

Returns:¶

tuple[str, …] – The keys of the JSON object. The keys are in the format of “key1.key2;index1.key3” where the semicolon indicates a list and the period indicates a nested object.
list[Any] – The values of the JSON object. The values are in the same order as the keys.

Return type:¶

tuple[tuple[str, …], list[Any]]

splatnet3_scraper.utils.match_partial_path(data, partial_path)¶

Returns a list of all paths in the given data that match the given partial path. For example, if partial_path is (0, "key1"), this will return all paths in the data that match ...[0]["key1"]. If fed a list of partial paths, this will return all paths that match any of the partial paths. Do not confuse tuples with lists, as they are treated differently.

The “:” string can be added to the partial path input to match all list indices that match it. This is useful for JSONs with a repeating structure.

The match_partial_path algorithm searches for all paths in a dictionary that match the given partial path. The partial_path can be a string, integer, a special “:” string, or a tuple of strings/integers that represents the path to an item in the dictionary. For example, the path ("key1", "key2", 2) corresponds to ...["key1"]["key2"][2] in the dictionary. The algorithm returns a list of all paths that match the partial_path. Each path in the result is represented as a tuple of strings and integers where strings correspond to dictionary keys and integers correspond to list indices.

For instance, if data is:

>>> data = {
...     "key1": {
...         "key2": [
...             {"key3": 1},
...             {"key3": 2},
...         ]
...     },
...     "key4": {
...         "key5": [
...             {"key3": 3},
...             {"key3": 4},
...         ]
...     }
... }

Then match_partial_path(data, (0, "key3")) will return:

>>> [
...     ("key1", "key2", 0, "key3"),
...     ("key4", "key5", 0, "key3"),
... ]

If match_partial_path(data, "key3") is called, the result will be:

>>> [
...     ("key1", "key2", 0, "key3"),
...     ("key1", "key2", 1, "key3"),
...     ("key4", "key5", 0, "key3"),
...     ("key4", "key5", 1, "key3"),
... ]

If match_partial_path(data, [(0, "key3"), "key2"]) is called, the result will be:

>>> [
...     ("key1", "key2", 0, "key3"),
...     ("key4", "key5", 0, "key3"),
...     ("key1", "key2"),
... ]

If match_partial_path(data, ("key1", "key2", ":", "key3")) is called, the result will be:

>>> [
...     ("key1", "key2", 0, "key3"),
...     ("key1", "key2", 1, "key3"),
... ]

Parameters:¶

data : dict[str, Any] | list¶: The dictionary or list to search.
partial_path : PathType | list[PathType]¶: The partial path to match. If the partial path is a tuple, this function will treat it as a path. For example, a partial_path argument of (0, "key1") will be treated as ...[0]["key1"]. If the partial path is a string, this function will treat it as a key in a dictionary. Integers will be treated as indices in a list. If the partial path is a list, this function will return all paths that match any of the partial paths. The special “:” string can be used to match all list indices that match the rest of the partial path. This is useful to get all paths in a list that match a given partial path.

Returns:¶

list[tuple[str | int, …]] – A list of all paths that match the given partial path. Each path is represented as a tuple of strings and integers, where strings correspond to dictionary keys and integers correspond to list indices.

Return type:¶

list[tuple[str | int, …]]

splatnet3_scraper.utils.retry(times, exceptions=<class 'Exception'>, call_on_fail=None)¶

Decorator that retries a function a specified number of times if it raises a specific exception or tuple of exceptions.

Parameters:¶

times : int¶: Max number of times to retry the function before raising the exception.
exceptions : tuple[Type[Exception], ...] | Type[Exception]: Exception or tuple of exceptions to catch. Defaults to Exception.
call_on_fail : Callable[[], None] | None: Function to call if the function fails. If None, nothing will be called. Defaults to None.

Returns:¶

Callable[[Callable[P, T]], Callable[P, T]] – The decorated function, which will retry the function if it raises an exception.

Return type:¶

Callable[[Callable[[~P], T]], Callable[[~P], T]]