Snakemake conda env parameter is not taken from config.yaml file
I use a conda env that I create manually, not automatically using Snakemake. I do this to keep tighter version control.
Anyway, in my config.yaml I have the following line:
conda_env: '/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake'
Then, at the start of my Snakefile I read that variable (reading variables from config in your shell part does not seem to work, am I right?):
conda_env = config['conda_env']
Then in a shell part I hail said parameter like this:
rule rsem_quantify:
input:
os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
output:
os.path.join(analyzed_dir, '{sample}.genes.results'),
os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
threads: 8
shell:
'''
#!/bin/bash
source activate {conda_env}
rsem-calculate-expression
--paired-end
{input}
{rsem_ref_base}
{analyzed_dir}/{wildcards.sample}
--strandedness reverse
--num-threads {threads}
--star
--star-gzipped-read-file
--star-output-genome-bam
'''
Notice the {conda_env}
. Now this gives me the following error:
Could not find conda environment: None
You can list all discoverable environments with `conda info --envs`.
Now, if I change {conda_env}
for its parameter directly /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
, it does work! I don't have any trouble reading other parameters using this method (like rsem_ref_base
and analyzed_dir
in the example rule above.
What could be wrong here?
Highest regards,
Freek.
bioinformatics conda snakemake
add a comment |
I use a conda env that I create manually, not automatically using Snakemake. I do this to keep tighter version control.
Anyway, in my config.yaml I have the following line:
conda_env: '/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake'
Then, at the start of my Snakefile I read that variable (reading variables from config in your shell part does not seem to work, am I right?):
conda_env = config['conda_env']
Then in a shell part I hail said parameter like this:
rule rsem_quantify:
input:
os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
output:
os.path.join(analyzed_dir, '{sample}.genes.results'),
os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
threads: 8
shell:
'''
#!/bin/bash
source activate {conda_env}
rsem-calculate-expression
--paired-end
{input}
{rsem_ref_base}
{analyzed_dir}/{wildcards.sample}
--strandedness reverse
--num-threads {threads}
--star
--star-gzipped-read-file
--star-output-genome-bam
'''
Notice the {conda_env}
. Now this gives me the following error:
Could not find conda environment: None
You can list all discoverable environments with `conda info --envs`.
Now, if I change {conda_env}
for its parameter directly /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
, it does work! I don't have any trouble reading other parameters using this method (like rsem_ref_base
and analyzed_dir
in the example rule above.
What could be wrong here?
Highest regards,
Freek.
bioinformatics conda snakemake
add a comment |
I use a conda env that I create manually, not automatically using Snakemake. I do this to keep tighter version control.
Anyway, in my config.yaml I have the following line:
conda_env: '/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake'
Then, at the start of my Snakefile I read that variable (reading variables from config in your shell part does not seem to work, am I right?):
conda_env = config['conda_env']
Then in a shell part I hail said parameter like this:
rule rsem_quantify:
input:
os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
output:
os.path.join(analyzed_dir, '{sample}.genes.results'),
os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
threads: 8
shell:
'''
#!/bin/bash
source activate {conda_env}
rsem-calculate-expression
--paired-end
{input}
{rsem_ref_base}
{analyzed_dir}/{wildcards.sample}
--strandedness reverse
--num-threads {threads}
--star
--star-gzipped-read-file
--star-output-genome-bam
'''
Notice the {conda_env}
. Now this gives me the following error:
Could not find conda environment: None
You can list all discoverable environments with `conda info --envs`.
Now, if I change {conda_env}
for its parameter directly /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
, it does work! I don't have any trouble reading other parameters using this method (like rsem_ref_base
and analyzed_dir
in the example rule above.
What could be wrong here?
Highest regards,
Freek.
bioinformatics conda snakemake
I use a conda env that I create manually, not automatically using Snakemake. I do this to keep tighter version control.
Anyway, in my config.yaml I have the following line:
conda_env: '/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake'
Then, at the start of my Snakefile I read that variable (reading variables from config in your shell part does not seem to work, am I right?):
conda_env = config['conda_env']
Then in a shell part I hail said parameter like this:
rule rsem_quantify:
input:
os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
output:
os.path.join(analyzed_dir, '{sample}.genes.results'),
os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
threads: 8
shell:
'''
#!/bin/bash
source activate {conda_env}
rsem-calculate-expression
--paired-end
{input}
{rsem_ref_base}
{analyzed_dir}/{wildcards.sample}
--strandedness reverse
--num-threads {threads}
--star
--star-gzipped-read-file
--star-output-genome-bam
'''
Notice the {conda_env}
. Now this gives me the following error:
Could not find conda environment: None
You can list all discoverable environments with `conda info --envs`.
Now, if I change {conda_env}
for its parameter directly /rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake
, it does work! I don't have any trouble reading other parameters using this method (like rsem_ref_base
and analyzed_dir
in the example rule above.
What could be wrong here?
Highest regards,
Freek.
bioinformatics conda snakemake
bioinformatics conda snakemake
edited Jan 3 at 21:49
merv
25.8k676112
25.8k676112
asked Jan 3 at 15:03
FreekFreek
3881519
3881519
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
The pattern I use is to load variables into params
, so something along the lines of
rule rsem_quantify:
input:
os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
output:
os.path.join(analyzed_dir, '{sample}.genes.results'),
os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
params:
conda_env=config['conda_env']
threads: 8
shell:
'''
#!/bin/bash
source activate {params.conda_env}
rsem-calculate-expression
...
'''
Although, I'd also never do this with a conda environment, because Snakemake has conda environment management built-in. See this section in the docs on Integrated Package Management for details. This makes reproducibility much more manageable.
Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.
– Freek
Jan 4 at 9:21
Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....
– Freek
Jan 4 at 15:20
I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.
– Freek
Jan 7 at 10:25
@Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice ofsource activate...
whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on thePATH
. See the v4.4 release notes for details.
– merv
Jan 7 at 15:07
Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.
– Freek
Jan 8 at 15:40
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54024862%2fsnakemake-conda-env-parameter-is-not-taken-from-config-yaml-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The pattern I use is to load variables into params
, so something along the lines of
rule rsem_quantify:
input:
os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
output:
os.path.join(analyzed_dir, '{sample}.genes.results'),
os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
params:
conda_env=config['conda_env']
threads: 8
shell:
'''
#!/bin/bash
source activate {params.conda_env}
rsem-calculate-expression
...
'''
Although, I'd also never do this with a conda environment, because Snakemake has conda environment management built-in. See this section in the docs on Integrated Package Management for details. This makes reproducibility much more manageable.
Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.
– Freek
Jan 4 at 9:21
Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....
– Freek
Jan 4 at 15:20
I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.
– Freek
Jan 7 at 10:25
@Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice ofsource activate...
whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on thePATH
. See the v4.4 release notes for details.
– merv
Jan 7 at 15:07
Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.
– Freek
Jan 8 at 15:40
add a comment |
The pattern I use is to load variables into params
, so something along the lines of
rule rsem_quantify:
input:
os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
output:
os.path.join(analyzed_dir, '{sample}.genes.results'),
os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
params:
conda_env=config['conda_env']
threads: 8
shell:
'''
#!/bin/bash
source activate {params.conda_env}
rsem-calculate-expression
...
'''
Although, I'd also never do this with a conda environment, because Snakemake has conda environment management built-in. See this section in the docs on Integrated Package Management for details. This makes reproducibility much more manageable.
Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.
– Freek
Jan 4 at 9:21
Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....
– Freek
Jan 4 at 15:20
I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.
– Freek
Jan 7 at 10:25
@Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice ofsource activate...
whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on thePATH
. See the v4.4 release notes for details.
– merv
Jan 7 at 15:07
Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.
– Freek
Jan 8 at 15:40
add a comment |
The pattern I use is to load variables into params
, so something along the lines of
rule rsem_quantify:
input:
os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
output:
os.path.join(analyzed_dir, '{sample}.genes.results'),
os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
params:
conda_env=config['conda_env']
threads: 8
shell:
'''
#!/bin/bash
source activate {params.conda_env}
rsem-calculate-expression
...
'''
Although, I'd also never do this with a conda environment, because Snakemake has conda environment management built-in. See this section in the docs on Integrated Package Management for details. This makes reproducibility much more manageable.
The pattern I use is to load variables into params
, so something along the lines of
rule rsem_quantify:
input:
os.path.join(fastq_dir, '{sample}_R1_001.fastq.gz'),
os.path.join(fastq_dir, '{sample}_R2_001.fastq.gz')
output:
os.path.join(analyzed_dir, '{sample}.genes.results'),
os.path.join(analyzed_dir, '{sample}.STAR.genome.bam')
params:
conda_env=config['conda_env']
threads: 8
shell:
'''
#!/bin/bash
source activate {params.conda_env}
rsem-calculate-expression
...
'''
Although, I'd also never do this with a conda environment, because Snakemake has conda environment management built-in. See this section in the docs on Integrated Package Management for details. This makes reproducibility much more manageable.
edited Jan 3 at 21:52
answered Jan 3 at 21:47
mervmerv
25.8k676112
25.8k676112
Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.
– Freek
Jan 4 at 9:21
Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....
– Freek
Jan 4 at 15:20
I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.
– Freek
Jan 7 at 10:25
@Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice ofsource activate...
whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on thePATH
. See the v4.4 release notes for details.
– merv
Jan 7 at 15:07
Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.
– Freek
Jan 8 at 15:40
add a comment |
Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.
– Freek
Jan 4 at 9:21
Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....
– Freek
Jan 4 at 15:20
I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.
– Freek
Jan 7 at 10:25
@Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice ofsource activate...
whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on thePATH
. See the v4.4 release notes for details.
– merv
Jan 7 at 15:07
Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.
– Freek
Jan 8 at 15:40
Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.
– Freek
Jan 4 at 9:21
Thank you, I can use that although it is about as much code as adding the conda env in each shell: part. I was hoping for something global. I understand that I can use conda, perhaps I should but it sounds like such a waist if I can set up my environment once and be done with it. I guess using integrated conda is great for portability though. But I use the same env for all my runs.
– Freek
Jan 4 at 9:21
Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....
– Freek
Jan 4 at 15:20
Although with your method I can of course change the env in one place in the config so that is very nice. I will test this when RSEM finishes....
– Freek
Jan 4 at 15:20
I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.
– Freek
Jan 7 at 10:25
I thought it worked but it indeed seems to add the conda env nicely however... I get this error now: "/rst1/2017-0205_illuminaseq/scratch/swo-406/snakemake/etc/conda/activate.d/java_home.sh: line 1: JAVA_HOME: unbound variable" could this be related? If not ill mark as working and start a new question.
– Freek
Jan 7 at 10:25
@Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice of
source activate...
whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on the PATH
. See the v4.4 release notes for details.– merv
Jan 7 at 15:07
@Freek that's a conda activation issue. Probably better searching/posting as a separate issue. Though, I will note that it seems you are following the older practice of
source activate...
whereas a more recent best practice was introduced in conda 4.4, which also includes changes to how you manage getting conda on the PATH
. See the v4.4 release notes for details.– merv
Jan 7 at 15:07
Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.
– Freek
Jan 8 at 15:40
Ok, I gave up and started using the build in conda dependencies management (with environment.yaml files) which is pretty great and reproducible as well because I can specify the exact versions of my tools, I found.
– Freek
Jan 8 at 15:40
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54024862%2fsnakemake-conda-env-parameter-is-not-taken-from-config-yaml-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown