2026-03-15 07:24:11,929 - distributed.worker - ERROR - Compute Failed Key: ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 53) State: executing Task: <Task ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 53) <dask.dataframe.io.parquet.core.ParquetFunctionWra(...)> Exception: "FileNotFoundError(2, 'No such file or directory')" Traceback: ' File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 82, in __call__\n return read_parquet_part(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 184, in read_parquet_part\n func(\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 572, in read_partition\n arrow_table = cls._read_table(\n ^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 1704, in _read_table\n arrow_table = _read_table_from_path(\n ^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 255, in _read_table_from_path\n with _open_input_files(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/utils.py", line 221, in _open_input_files\n return [_set_context(fs.open(path, **kwargs), context_stack) for path in paths]\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/spec.py", line 1338, in open\n f = self._open(\n ^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 210, in _open\n return LocalFileOpener(path, mode, fs=self, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 387, in __init__\n self._open()\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 392, in _open\n self.f = open(self.path, mode=self.mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'

2026-03-15 07:24:11,923 - distributed.worker - ERROR - Compute Failed Key: ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 359) State: executing Task: <Task ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 359) <dask.dataframe.io.parquet.core.ParquetFunctionWra(...)> Exception: "FileNotFoundError(2, 'No such file or directory')" Traceback: ' File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 82, in __call__\n return read_parquet_part(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 184, in read_parquet_part\n func(\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 572, in read_partition\n arrow_table = cls._read_table(\n ^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 1704, in _read_table\n arrow_table = _read_table_from_path(\n ^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 255, in _read_table_from_path\n with _open_input_files(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/utils.py", line 221, in _open_input_files\n return [_set_context(fs.open(path, **kwargs), context_stack) for path in paths]\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/spec.py", line 1338, in open\n f = self._open(\n ^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 210, in _open\n return LocalFileOpener(path, mode, fs=self, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 387, in __init__\n self._open()\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 392, in _open\n self.f = open(self.path, mode=self.mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'

2026-03-15 07:21:14,805 - distributed.worker - ERROR - Compute Failed Key: ('read_parquet-8966918c2c258d0f3de779f8a5fd24b5', 495) State: executing Task: <Task ('read_parquet-8966918c2c258d0f3de779f8a5fd24b5', 495) <dask.dataframe.io.parquet.core.ParquetFunctionWra(...)> Exception: "FileNotFoundError(2, 'No such file or directory')" Traceback: ' File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 82, in __call__\n return read_parquet_part(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 184, in read_parquet_part\n func(\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 572, in read_partition\n arrow_table = cls._read_table(\n ^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 1704, in _read_table\n arrow_table = _read_table_from_path(\n ^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 255, in _read_table_from_path\n with _open_input_files(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/utils.py", line 221, in _open_input_files\n return [_set_context(fs.open(path, **kwargs), context_stack) for path in paths]\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/spec.py", line 1338, in open\n f = self._open(\n ^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 210, in _open\n return LocalFileOpener(path, mode, fs=self, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 387, in __init__\n self._open()\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 392, in _open\n self.f = open(self.path, mode=self.mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'

2026-03-15 07:21:14,805 - distributed.worker - ERROR - Compute Failed Key: ('read_parquet-8966918c2c258d0f3de779f8a5fd24b5', 476) State: executing Task: <Task ('read_parquet-8966918c2c258d0f3de779f8a5fd24b5', 476) <dask.dataframe.io.parquet.core.ParquetFunctionWra(...)> Exception: "FileNotFoundError(2, 'No such file or directory')" Traceback: ' File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 82, in __call__\n return read_parquet_part(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 184, in read_parquet_part\n func(\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 572, in read_partition\n arrow_table = cls._read_table(\n ^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 1704, in _read_table\n arrow_table = _read_table_from_path(\n ^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 255, in _read_table_from_path\n with _open_input_files(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/utils.py", line 221, in _open_input_files\n return [_set_context(fs.open(path, **kwargs), context_stack) for path in paths]\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/spec.py", line 1338, in open\n f = self._open(\n ^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 210, in _open\n return LocalFileOpener(path, mode, fs=self, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 387, in __init__\n self._open()\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 392, in _open\n self.f = open(self.path, mode=self.mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'

2026-03-15 07:17:43,084 - distributed.worker - ERROR - Compute Failed Key: ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 345) State: executing Task: <Task ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 345) <dask.dataframe.io.parquet.core.ParquetFunctionWra(...)> Exception: "FileNotFoundError(2, 'No such file or directory')" Traceback: ' File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 82, in __call__\n return read_parquet_part(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 184, in read_parquet_part\n func(\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 572, in read_partition\n arrow_table = cls._read_table(\n ^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 1704, in _read_table\n arrow_table = _read_table_from_path(\n ^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 255, in _read_table_from_path\n with _open_input_files(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/utils.py", line 221, in _open_input_files\n return [_set_context(fs.open(path, **kwargs), context_stack) for path in paths]\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/spec.py", line 1338, in open\n f = self._open(\n ^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 210, in _open\n return LocalFileOpener(path, mode, fs=self, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 387, in __init__\n self._open()\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 392, in _open\n self.f = open(self.path, mode=self.mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'

2026-03-15 07:17:43,084 - distributed.worker - ERROR - Compute Failed Key: ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 72) State: executing Task: <Task ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 72) <dask.dataframe.io.parquet.core.ParquetFunctionWra(...)> Exception: "FileNotFoundError(2, 'No such file or directory')" Traceback: ' File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 82, in __call__\n return read_parquet_part(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 184, in read_parquet_part\n func(\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 572, in read_partition\n arrow_table = cls._read_table(\n ^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 1704, in _read_table\n arrow_table = _read_table_from_path(\n ^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 255, in _read_table_from_path\n with _open_input_files(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/utils.py", line 221, in _open_input_files\n return [_set_context(fs.open(path, **kwargs), context_stack) for path in paths]\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/spec.py", line 1338, in open\n f = self._open(\n ^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 210, in _open\n return LocalFileOpener(path, mode, fs=self, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 387, in __init__\n self._open()\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 392, in _open\n self.f = open(self.path, mode=self.mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'

2026-03-15 07:17:43,084 - distributed.worker - ERROR - Compute Failed Key: ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 485) State: executing Task: <Task ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 485) <dask.dataframe.io.parquet.core.ParquetFunctionWra(...)> Exception: "FileNotFoundError(2, 'No such file or directory')" Traceback: ' File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 82, in __call__\n return read_parquet_part(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 184, in read_parquet_part\n func(\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 572, in read_partition\n arrow_table = cls._read_table(\n ^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 1704, in _read_table\n arrow_table = _read_table_from_path(\n ^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 255, in _read_table_from_path\n with _open_input_files(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/utils.py", line 221, in _open_input_files\n return [_set_context(fs.open(path, **kwargs), context_stack) for path in paths]\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/spec.py", line 1338, in open\n f = self._open(\n ^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 210, in _open\n return LocalFileOpener(path, mode, fs=self, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 387, in __init__\n self._open()\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 392, in _open\n self.f = open(self.path, mode=self.mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'

2026-03-15 07:17:43,083 - distributed.worker - ERROR - Compute Failed Key: ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 364) State: executing Task: <Task ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 364) <dask.dataframe.io.parquet.core.ParquetFunctionWra(...)> Exception: "FileNotFoundError(2, 'No such file or directory')" Traceback: ' File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 82, in __call__\n return read_parquet_part(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 184, in read_parquet_part\n func(\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 572, in read_partition\n arrow_table = cls._read_table(\n ^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 1704, in _read_table\n arrow_table = _read_table_from_path(\n ^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 255, in _read_table_from_path\n with _open_input_files(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/utils.py", line 221, in _open_input_files\n return [_set_context(fs.open(path, **kwargs), context_stack) for path in paths]\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/spec.py", line 1338, in open\n f = self._open(\n ^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 210, in _open\n return LocalFileOpener(path, mode, fs=self, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 387, in __init__\n self._open()\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 392, in _open\n self.f = open(self.path, mode=self.mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'

2026-03-15 07:17:43,069 - distributed.worker - ERROR - Compute Failed Key: ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 359) State: executing Task: <Task ('read_parquet-d002924f5c8ac6995f77a96974b4a56a', 359) <dask.dataframe.io.parquet.core.ParquetFunctionWra(...)> Exception: "FileNotFoundError(2, 'No such file or directory')" Traceback: ' File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 82, in __call__\n return read_parquet_part(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/core.py", line 184, in read_parquet_part\n func(\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 572, in read_partition\n arrow_table = cls._read_table(\n ^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 1704, in _read_table\n arrow_table = _read_table_from_path(\n ^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/parquet/arrow.py", line 255, in _read_table_from_path\n with _open_input_files(\n ^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/dask/dataframe/io/utils.py", line 221, in _open_input_files\n return [_set_context(fs.open(path, **kwargs), context_stack) for path in paths]\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/spec.py", line 1338, in open\n f = self._open(\n ^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 210, in _open\n return LocalFileOpener(path, mode, fs=self, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 387, in __init__\n self._open()\n File "/opt/conda/lib/python3.12/site-packages/fsspec/implementations/local.py", line 392, in _open\n self.f = open(self.path, mode=self.mode)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'

2026-02-20 11:06:10,935 - distributed.worker - INFO - -------------------------------------------------

2026-02-20 11:06:10,935 - distributed.worker - INFO - Registered to: tcp://simple-scheduler.dask-operator.svc.cluster.local:8786

2026-02-20 11:06:10,934 - distributed.worker - INFO - Starting Worker plugin shuffle

2026-02-20 11:06:10,906 - distributed.worker - INFO - -------------------------------------------------

2026-02-20 11:06:10,906 - distributed.worker - INFO - Local Directory: /mnt/scratch/dask-scratch-space/worker-n1xcl_6f

2026-02-20 11:06:10,906 - distributed.worker - INFO - Memory: 70.00 GiB

2026-02-20 11:06:10,906 - distributed.worker - INFO - Threads: 5

2026-02-20 11:06:10,906 - distributed.worker - INFO - -------------------------------------------------

2026-02-20 11:06:10,906 - distributed.worker - INFO - Waiting to connect to: tcp://simple-scheduler.dask-operator.svc.cluster.local:8786

2026-02-20 11:06:10,906 - distributed.worker - INFO - dashboard at: 172.16.162.210:8788

2026-02-20 11:06:10,906 - distributed.worker - INFO - Worker name: simple-default-worker-dfc6a92879

2026-02-20 11:06:10,906 - distributed.worker - INFO - Listening to: tcp://172.16.162.210:38221

2026-02-20 11:06:10,905 - distributed.worker - INFO - Start worker at: tcp://172.16.162.210:38221