Snakemake conda env parameter is not taken from config.yaml file












0















I use a conda env that I create manually, not automatically using Snakemake. I do this to keep tighter version control.



Anyway, in my config.yaml I have the following line:



conda_env: '/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake'


Then, at the start of my Snakefile I read that variable (reading variables from config in your shell part does not seem to work, am I right?):



conda_env = config['conda_env']


Then in a shell part I hail said parameter like this:



rule rsem_quantify:
input:
os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
output:
os.path.join(analyzed_dir, '{sample}.genes.results'),
os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
threads: 8
shell:
'''
#!/bin/bash
source activate {conda_env}

rsem-calculate-expression
--paired-end
{input}
{rsem_ref_base}
{analyzed_dir}/{wildcards.sample}
--strandedness reverse
--num-threads {threads}
--star
--star-gzipped-read-file
--star-output-genome-bam
'''


Notice the {conda_env}. Now this gives me the following error:



Could not find conda environment: None
You can list all discoverable environments with `conda info --envs`.


Now, if I change {conda_env} for its parameter directly /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake, it does work! I don't have any trouble reading other parameters using this method (like rsem_ref_base and analyzed_dir in the example rule above.



What could be wrong here?



Highest regards,



Freek.










share|improve this question





























    0















    I use a conda env that I create manually, not automatically using Snakemake. I do this to keep tighter version control.



    Anyway, in my config.yaml I have the following line:



    conda_env: '/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake'


    Then, at the start of my Snakefile I read that variable (reading variables from config in your shell part does not seem to work, am I right?):



    conda_env = config['conda_env']


    Then in a shell part I hail said parameter like this:



    rule rsem_quantify:
    input:
    os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
    os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
    output:
    os.path.join(analyzed_dir, '{sample}.genes.results'),
    os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
    threads: 8
    shell:
    '''
    #!/bin/bash
    source activate {conda_env}

    rsem-calculate-expression
    --paired-end
    {input}
    {rsem_ref_base}
    {analyzed_dir}/{wildcards.sample}
    --strandedness reverse
    --num-threads {threads}
    --star
    --star-gzipped-read-file
    --star-output-genome-bam
    '''


    Notice the {conda_env}. Now this gives me the following error:



    Could not find conda environment: None
    You can list all discoverable environments with `conda info --envs`.


    Now, if I change {conda_env} for its parameter directly /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake, it does work! I don't have any trouble reading other parameters using this method (like rsem_ref_base and analyzed_dir in the example rule above.



    What could be wrong here?



    Highest regards,



    Freek.










    share|improve this question



























      0












      0








      0








      I use a conda env that I create manually, not automatically using Snakemake. I do this to keep tighter version control.



      Anyway, in my config.yaml I have the following line:



      conda_env: '/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake'


      Then, at the start of my Snakefile I read that variable (reading variables from config in your shell part does not seem to work, am I right?):



      conda_env = config['conda_env']


      Then in a shell part I hail said parameter like this:



      rule rsem_quantify:
      input:
      os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
      os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
      output:
      os.path.join(analyzed_dir, '{sample}.genes.results'),
      os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
      threads: 8
      shell:
      '''
      #!/bin/bash
      source activate {conda_env}

      rsem-calculate-expression
      --paired-end
      {input}
      {rsem_ref_base}
      {analyzed_dir}/{wildcards.sample}
      --strandedness reverse
      --num-threads {threads}
      --star
      --star-gzipped-read-file
      --star-output-genome-bam
      '''


      Notice the {conda_env}. Now this gives me the following error:



      Could not find conda environment: None
      You can list all discoverable environments with `conda info --envs`.


      Now, if I change {conda_env} for its parameter directly /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake, it does work! I don't have any trouble reading other parameters using this method (like rsem_ref_base and analyzed_dir in the example rule above.



      What could be wrong here?



      Highest regards,



      Freek.










      share|improve this question
















      I use a conda env that I create manually, not automatically using Snakemake. I do this to keep tighter version control.



      Anyway, in my config.yaml I have the following line:



      conda_env: '/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake'


      Then, at the start of my Snakefile I read that variable (reading variables from config in your shell part does not seem to work, am I right?):



      conda_env = config['conda_env']


      Then in a shell part I hail said parameter like this:



      rule rsem_quantify:
      input:
      os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
      os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
      output:
      os.path.join(analyzed_dir, '{sample}.genes.results'),
      os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
      threads: 8
      shell:
      '''
      #!/bin/bash
      source activate {conda_env}

      rsem-calculate-expression
      --paired-end
      {input}
      {rsem_ref_base}
      {analyzed_dir}/{wildcards.sample}
      --strandedness reverse
      --num-threads {threads}
      --star
      --star-gzipped-read-file
      --star-output-genome-bam
      '''


      Notice the {conda_env}. Now this gives me the following error:



      Could not find conda environment: None
      You can list all discoverable environments with `conda info --envs`.


      Now, if I change {conda_env} for its parameter directly /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake, it does work! I don't have any trouble reading other parameters using this method (like rsem_ref_base and analyzed_dir in the example rule above.



      What could be wrong here?



      Highest regards,



      Freek.







      bioinformatics conda snakemake






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Jan 3 at 21:49









      merv

      25.8k676112




      25.8k676112










      asked Jan 3 at 15:03









      FreekFreek

      3881519




      3881519
























          1 Answer
          1






          active

          oldest

          votes


















          2














          The pattern I use is to load variables into params, so something along the lines of



          rule rsem_quantify:
          input:
          os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
          os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
          output:
          os.path.join(analyzed_dir, '{sample}.genes.results'),
          os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
          params:
          conda_env=config['conda_env']
          threads: 8
          shell:
          '''
          #!/bin/bash
          source activate {params.conda_env}

          rsem-calculate-expression

          ...

          '''


          Although, I'd also never do this with a conda environment, because Snakemake has conda environment management built-in. See this section in the docs on Integrated Package Management for details. This makes reproducibility much more manageable.






          share|improve this answer


























          • Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.

            – Freek
            Jan 4 at 9:21













          • Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....

            – Freek
            Jan 4 at 15:20











          • I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.

            – Freek
            Jan 7 at 10:25











          • @Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice of source activate... whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on the PATH. See the v4.4 release notes for details.

            – merv
            Jan 7 at 15:07











          • Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.

            – Freek
            Jan 8 at 15:40












          Your Answer






          StackExchange.ifUsing("editor", function () {
          StackExchange.using("externalEditor", function () {
          StackExchange.using("snippets", function () {
          StackExchange.snippets.init();
          });
          });
          }, "code-snippets");

          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "1"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














          draft saved

          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54024862%2fsnakemake-conda-env-parameter-is-not-taken-from-config-yaml-file%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          2














          The pattern I use is to load variables into params, so something along the lines of



          rule rsem_quantify:
          input:
          os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
          os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
          output:
          os.path.join(analyzed_dir, '{sample}.genes.results'),
          os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
          params:
          conda_env=config['conda_env']
          threads: 8
          shell:
          '''
          #!/bin/bash
          source activate {params.conda_env}

          rsem-calculate-expression

          ...

          '''


          Although, I'd also never do this with a conda environment, because Snakemake has conda environment management built-in. See this section in the docs on Integrated Package Management for details. This makes reproducibility much more manageable.






          share|improve this answer


























          • Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.

            – Freek
            Jan 4 at 9:21













          • Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....

            – Freek
            Jan 4 at 15:20











          • I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.

            – Freek
            Jan 7 at 10:25











          • @Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice of source activate... whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on the PATH. See the v4.4 release notes for details.

            – merv
            Jan 7 at 15:07











          • Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.

            – Freek
            Jan 8 at 15:40
















          2














          The pattern I use is to load variables into params, so something along the lines of



          rule rsem_quantify:
          input:
          os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
          os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
          output:
          os.path.join(analyzed_dir, '{sample}.genes.results'),
          os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
          params:
          conda_env=config['conda_env']
          threads: 8
          shell:
          '''
          #!/bin/bash
          source activate {params.conda_env}

          rsem-calculate-expression

          ...

          '''


          Although, I'd also never do this with a conda environment, because Snakemake has conda environment management built-in. See this section in the docs on Integrated Package Management for details. This makes reproducibility much more manageable.






          share|improve this answer


























          • Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.

            – Freek
            Jan 4 at 9:21













          • Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....

            – Freek
            Jan 4 at 15:20











          • I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.

            – Freek
            Jan 7 at 10:25











          • @Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice of source activate... whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on the PATH. See the v4.4 release notes for details.

            – merv
            Jan 7 at 15:07











          • Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.

            – Freek
            Jan 8 at 15:40














          2












          2








          2







          The pattern I use is to load variables into params, so something along the lines of



          rule rsem_quantify:
          input:
          os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
          os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
          output:
          os.path.join(analyzed_dir, '{sample}.genes.results'),
          os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
          params:
          conda_env=config['conda_env']
          threads: 8
          shell:
          '''
          #!/bin/bash
          source activate {params.conda_env}

          rsem-calculate-expression

          ...

          '''


          Although, I'd also never do this with a conda environment, because Snakemake has conda environment management built-in. See this section in the docs on Integrated Package Management for details. This makes reproducibility much more manageable.






          share|improve this answer















          The pattern I use is to load variables into params, so something along the lines of



          rule rsem_quantify:
          input:
          os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
          os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
          output:
          os.path.join(analyzed_dir, '{sample}.genes.results'),
          os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
          params:
          conda_env=config['conda_env']
          threads: 8
          shell:
          '''
          #!/bin/bash
          source activate {params.conda_env}

          rsem-calculate-expression

          ...

          '''


          Although, I'd also never do this with a conda environment, because Snakemake has conda environment management built-in. See this section in the docs on Integrated Package Management for details. This makes reproducibility much more manageable.







          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited Jan 3 at 21:52

























          answered Jan 3 at 21:47









          mervmerv

          25.8k676112




          25.8k676112













          • Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.

            – Freek
            Jan 4 at 9:21













          • Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....

            – Freek
            Jan 4 at 15:20











          • I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.

            – Freek
            Jan 7 at 10:25











          • @Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice of source activate... whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on the PATH. See the v4.4 release notes for details.

            – merv
            Jan 7 at 15:07











          • Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.

            – Freek
            Jan 8 at 15:40



















          • Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.

            – Freek
            Jan 4 at 9:21













          • Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....

            – Freek
            Jan 4 at 15:20











          • I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.

            – Freek
            Jan 7 at 10:25











          • @Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice of source activate... whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on the PATH. See the v4.4 release notes for details.

            – merv
            Jan 7 at 15:07











          • Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.

            – Freek
            Jan 8 at 15:40

















          Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.

          – Freek
          Jan 4 at 9:21







          Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.

          – Freek
          Jan 4 at 9:21















          Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....

          – Freek
          Jan 4 at 15:20





          Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....

          – Freek
          Jan 4 at 15:20













          I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.

          – Freek
          Jan 7 at 10:25





          I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.

          – Freek
          Jan 7 at 10:25













          @Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice of source activate... whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on the PATH. See the v4.4 release notes for details.

          – merv
          Jan 7 at 15:07





          @Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice of source activate... whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on the PATH. See the v4.4 release notes for details.

          – merv
          Jan 7 at 15:07













          Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.

          – Freek
          Jan 8 at 15:40





          Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.

          – Freek
          Jan 8 at 15:40




















          draft saved

          draft discarded




















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid



          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54024862%2fsnakemake-conda-env-parameter-is-not-taken-from-config-yaml-file%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          Monofisismo

          Angular Downloading a file using contenturl with Basic Authentication

          Olmecas