IoPathFileLister

class torchdata.datapipes.iter.IoPathFileLister(root: str | Sequence[str] | IterDataPipe, masks: str | List[str] = '', *, pathmgr=None)

Lists the contents of the directory at the provided root pathname or URL, and yields the full pathname or URL for each file within the directory (functional name: list_files_by_iopath).

Parameters:

• root – The root local filepath or URL directory or list of roots to list files from

• masks – Unix style filter string or string list for filtering file name(s)

• pathmgr – Custom iopath.PathManager. If not specified, a default PathManager is created.

Note

Default PathManager currently supports local file path, normal HTTP URL and OneDrive URL. S3 URL is supported only with iopath>=0.1.9.

Example

>>> from torchdata.datapipes.iter import IoPathFileLister
>>> datapipe = IoPathFileLister(root=S3URL)