WebHDFS

WebHDFS provides a complete FileSystem interface for HDFS over HTTP. Supports also HttpFS gateways. omniload supports WebHDFS as a data source.

URI Format

The URI for connecting to WebHDFS is structured as follows. Either access the resource anonymously, use key/secret credentials, or a security token for authentication. In this form, the remote resource is addressed exclusively using a single parameter, an URI, encoding all required options.

webhdfs://host:9870/path

URI parameters

host:

str Name-node address.

port:

int Port for webHDFS server.

kerberos:

Whether to authenticate with kerberos for this connection. Type: bool. Default: false.

token:

If given, use this token on every call to authenticate. A user and user-proxy may be encoded in the token and should not be also given.

user:

If given, assert the user name to connect with.

password:

If given, assert the password to use for basic auth. If password is provided, user must be provided also.

proxy_to:

If given, the user has the authority to proxy, and this value is the user in who’s name actions are taken.

kerberos_options:

Any extra arguments for HTTPKerberosAuth, see kerberos_.py. Type: dict. Use JSON to encode the dictionary.

data_proxy:

Map host names / data-node addresses host->data_proxy[host]. This can be necessary if the HDFS cluster is behind a proxy, running on Docker or otherwise has a mismatch between the host-names given by the name-node and the address by which to refer to them from the client. Type: dict. Use JSON to encode the dictionary.

use_https:

Whether to connect to the Name-node using HTTPS instead of HTTP. Type: bool. Default: false.

session_cert:

Path to a certificate file, or tuple of (cert, key) files to use for the conversation. Remark: Decoding tuples is not implemented yet.

session_verify:

Whether to verify the conversation, or the path to a certificate file for doing so. Type: str|bool. Default: true.

Authentication

Four authentication mechanisms are supported.

insecure:

No authentication is performed, and the user is assumed to be whoever they say they are (parameter user), or a predefined value such as “dr.who” if not given.

spnego:

When kerberos authentication is enabled, authentication is negotiated by requests_kerberos. This establishes a session based on existing kinit login and/or specified principal/password; parameters are passed with kerberos_options.

token:

Uses an existing Hadoop delegation token from another secured service. Indeed, this client can also generate such tokens when not insecure. Note that tokens expire, but can be renewed (by a previously specified user) and may allow for proxying.

basic-auth:

Used when both parameter user and parameter password are provided.

Examples

Load Parquet data from WebHDFS into DuckDB

The following command demonstrates how to copy data from a specified OSS location into a DuckDB database.

omniload ingest \
    --source-uri   'webhdfs://host:9870/path' \
    --source-table 'path/to/data.parquet' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.data'

Running the command creates a table named data within the testdrive schema in the DuckDB database file located at demo.duckdb.