Bdtjtk

I have written a docker file which adds my python script inside the container:
ADD test_pclean.py /test_pclean.py

My directory structure is:

.

├── Dockerfile

├── README.md

├── pipeline.json

└── test_pclean.py

My json file which acts as a configuration file for creating a pipeline in Pachyderm is as follows:

{

    "pipeline": {

        "name": "mopng-beneficiary-v2"

    },

    "transform": {

        "cmd": ["python3", "/test_pclean.py"],

        "image": "avisrivastava254084/mopng-beneficiary-v2-image-7"

    },

    "input": {

        "atom": {

            "repo": "mopng_beneficiary_v2",

            "glob": "/*"

        }

    }

}

Even though I have copied the official documentation's example, I am facing an error:
python3: can't open file '/test_pclean.py': [Errno 2] No such file or directory

My dockerfile is:

FROM    debian:stretch



# Install opencv and matplotlib.

RUN apt-get update 

    && apt-get upgrade -y 

    && apt-get install -y unzip wget build-essential 

        cmake git pkg-config libswscale-dev 

    && apt-get clean 

    && rm -rf /var/lib/apt



RUN apt update

RUN apt-get -y install python3-pip

RUN pip3 install matplotlib

RUN pip3 install pandas



ADD test_pclean.py /test_pclean.py

ENTRYPOINT [ "/bin/bash/" ]

Like some of the comments above suggest. It looks like your test_pclean.py file isn't in the docker image. Here's what should fix it.

1. Make sure your test_pclean.py file is in your docker image by having be included as part of the build process. Put this as the last step in your dockerfile:

COPY test_pclean.py .

2. Ensure that your pachyderm pipeline spec has the following for the cmd portion:
   "cmd": ["python3", "./test_pclean.py"]




3. And this is more of a suggestion than a requirement.... You'll make life easier for yourself if you use image tags as part of your docker build. If you default to latest tag, any future iterations/builds of this step in your pipeline could have negitave affects (new bugs in your code etc.). Therefore the best practice is to use a particular version in your pipeline: mopng-beneficiary-v2-image-7:v1 and mopng-beneficiary-v2-image-7:v2 and so on. That way you can iterate on say version 3 and it won't affect the already running pipeline.

docker build -t avisrivastava254084/mopng-beneficiary-v2-image-7:v1

Then just update your pipeline spec to use avisrivastava254084/mopng-beneficiary-v2-image-7:v1

I was not changing the commits to my docker images on each build and hence, Kubernetes was using the local docker file that it had(w/o tags and commits, it doesn't acknowledge any change). Once I started using commit with each build, Kubernetes started downloading the intended docker image.