Category: Linux

Categories
container emacs

Containerized Jupyter Lab Authentication for Emacs-IPython-Notebook

In my earlier post, I mentioned how to run Apache Spark in a container with Jupyter notebook. While I like to use Jupyter notebook to document my work and share it with my colleagues, I still prefer to use Emacs instead of a web browser to write code. The answer is EIN — Emacs IPython Notebook. Installing EIN is straight forward. The tricky part is to make it working with the containerized Jupyter.

Once you enter EIN, you will need to use the command “ein:login” to connect to Jupyter. Internally it goes through websocket and if you just use your password, as you would from a browser, EIN will throw an error of expired websocket and you will not be able to execute any code or create a new notebook. One fix of this problem is to use the Jupyter token as the password. The token can be found from the command below:

host$ docker exec spark-jupyter jupyter server list
Currently running servers:
http://1e2480237424:8888/?token=4899803e93c22739a8de56fa4deed22aef6568f93025c901 :: /home/jovyan

Jupyter can use both token and password for user authentication based on how it is configured. By default, token changes every time, which makes it more secure but also harder to use. In a secured environment, we can fix the token by define the JUPYTER_TOKEN environment (first line below) and pass it to the Docker container when starting Jupyter (second line):

export JUPYTER_TOKEN="4899803e93c22739a8de56fa4deed22aef6568f93025c901"

host$ docker run --name spark -d -p 8888:8888 -p 4040:4040 -p 4041:4041 -e JUPYTER_TOKEN=$JUPYTER_TOKEN -v $PWD:/home/jovyan --name spark-jupyter jupyter/all-spark-notebook

A better configuration is to run the “jupyter server password” command (new in Jupyter version 5.0 ) in the container, which will save a hashed password in /home/jovyan/.ipython/jupyter_server_config.json. Then you will only need to put the same password when login. Though you can also disable both password and token, I would recommend not to do it.

To enable inline image display in EIN/Emacs, just put this line in your .emacs file:

(setq ein:output-area-inlined-images t)
Categories
Linux

Update R and Bioconductor

It’s not a straightforward thing to update R and Bioconductor (a bioinformatics package collection for R), especially if you used the R package that comes with your Linux distribution. For example, the R package from Ubuntu package repository is still 3.5, while the latest version from the r-project site is already 4.0.3. While using an older version of R itself for statistics may not be a problem, many of the Bioconductor packages do have updates and bug fixes that require R v4. And to make things worse, updating Bioconductor itself is a painful process. Below is what I did to update R, Bioconductor, and associated packages. The main lesson learned here is not to use the R package from Ubuntu repository.

  1. Remove R installed from the Ubuntu repository;
  2. Install R from r-project by adding it as an apt repository;
  3. Update R packages (system-wide);
  4. Update R packages installed in the user directory;
  5. Update BiocManager
sudo apt-get purge r-base* r-recommended r-cran-*
sudo apt autoremove
sudo apt update
sudo add-apt-repository 'deb https://cloud.r-project.org/bin/linux/ubuntu focal-cran40/'
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
sudo apt update
sudo apt install r-base r-base-core r-recommended r-base-dev
update.packages(ask = FALSE, checkBuilt = TRUE)
if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")
BiocManager::install(version = "3.12")

The R code section may need to run twice once as a sudo user and once as a normal user if you also have packages installed under a user name (that is to say not a system-wide installation from system admin). The last command tells BiocManager (Bioconductor package manager) to install Bioconductor version 3.12 and update all installed Bioconductor pages.

Categories
emacs Linux

Configuration for Developing R code with ESS

It has been close to 10 years since I use R seriously last time. Back then I used ESS in Emacs to do all the editing and interactive sessions. I really liked it back then and I would like to check it out how ESS has evolved through the years. A quick glimpse of the ESS manual suggests it has got a lot of improvements and the Bioconductor project has evolved as well. For example, even the package installation package has changed in Bioconductor.

Unfortunately, I have forgotten the tricks and configurations that I have used to set up my ESS and Emacs environment. Below are a few things that I had to go through.

$sudo apt install -y r-cran-devtools
$sudo apt install libcurl4-gnutls-dev

The above two packages are required for quite some other R packages. To make sure the user library path is loaded correctly, I had to set the following environmental variable:

export R_LIBS="~/.local/R/lib"

The above setting works for starting R in Terminal. To make the local R library (in my home directory with no root privileges required) work with the ESS R session, I had to create a ~/.Rprofile file with the following content:

.libPaths("~/.local/R/lib")

Now my Emacs speaks ESS well. I can install packages to my local libraries by default and use Emacs to edit R code in one frame and run an interactive R session in another frame and use “C-c C-c” to execute a selected region in the R code frame.

  • Tags
Categories
emacs Linux

Setup NCBI E-utilities

The E-utilities toolbox from NCBI is the recommended way to access NCBI and NLM data, including the genome data. The following command can easily install it in the Ubuntu Linux distribution.

$sudo apt update
$sudo apt install ncbi-entrez-direct

After successful installation you can test it out using the following example code, which provides ftp path to the first 10 Bacillus cereus in refseq. Unfortunately, this code needs to run in the BASH shell. Running it in eshell (within Emacs) does not work.

$esearch -db assembly -query "Bacillus cereus " | efetch -format docsum -mode json|grep "ftppath_refseq" |head -10
            "ftppath_refseq": "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/013/394/245/GCF_013394245.1_ASM1339424v1",
            "ftppath_refseq": "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/013/303/115/GCF_013303115.1_ASM1330311v1",
            "ftppath_refseq": "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/013/284/505/GCF_013284505.1_ASM1328450v1",
            "ftppath_refseq": "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/013/284/455/GCF_013284455.1_ASM1328445v1",
            "ftppath_refseq": "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/013/284/445/GCF_013284445.1_ASM1328444v1",
            "ftppath_refseq": "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/013/284/425/GCF_013284425.1_ASM1328442v1",
            "ftppath_refseq": "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/013/267/775/GCF_013267775.1_ASM1326777v1",
            "ftppath_refseq": "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/013/267/475/GCF_013267475.1_ASM1326747v1",
            "ftppath_refseq": "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/013/267/455/GCF_013267455.1_ASM1326745v1",
            "ftppath_refseq": "ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/013/267/275/GCF_013267275.1_ASM1326727v1",

Categories
emacs

View PDF in Emacs

Be default PDF files are opened by Docviewer in Ubuntu distributes. Yet, there is a more convenient PDF tool, aptly named pdf-tools that works well with Emacs. The installation of pdf-tools is straight forward. Below is what I did.

sudo apt install libpng-dev zlib1g-dev
sudo apt install libpoppler-glib-dev
sudo apt install libpoppler-private-dev

After that, go to Emacs and use the package-install to install

package-refresh-contents
package-install pdf-tools

Finally, add the following line to the .emacs file. And make sure that if you are displaying line numbers in emacs, make sure it is disabled in pdf-view-mode. Here what I used is to just enable line number display in prog-mode.

(pdf-tools-install)
(add-hook 'prog-mode-hook 'linum-on)

Categories
emacs

Switch Frame Quickly in Emacs

Switching between multiple frames in Emacs is a pain when you have more than two frames open. I just noticed that the ace-window package offers the function for fast frame switching. All you need to do is to put the following configuration to the .emacs file and use the “M-o” key combination then pickup the frame number in red, and you will switch to that frame.

(global-set-key (kbd "M-o") 'ace-window)
Categories
emacs

Open Virtual Box File through SSH with Emacs

The Emacs Dire mode works for remote files through SSH and FTP. Detailed documentation about how to open remote files from Emacs can be found here. With a proper setting, one can use Emacs to work directly to open files in Virtual Box without having to set the target directory as a shared resource between the host and client. In a previous post, I showed how to add the private key used by Vagrant to access Virtual Box and find out the port number for SSH connection, in this post, you will see how it will benefit you as an Emacs user.

/ssh:vagrant@127.0.0.1#2222:/vagrant:

Once you are in Emacs, use the command “Ctrl x + Ctrl f” to open file just like you would with a local file, then you use the command above to open the remote directory “/vagrant” in a Emacs frame. Of course, this command requires that you have already added the private key to your host SSH key settings and 2222 is the correct port number.

Categories
Linux

Update Kernel in Ubuntu

The Linux kernel (v5.4.0) used in Ubuntu 20.04 has poor support for the HDMI sound support—the sound device shows “Dummy Output” after every computer sleep. I had tried different solutions but still would like to forget about running those commands before getting the sound out. So, I upgraded the kernel to the latest version supported by Ubuntu and it works. Below is the list of commands that I used.

$ wget https://raw.githubusercontent.com/pimlie/ubuntu-mainline-kernel.sh/master/ubuntu-mainline-kernel.sh
$ sudo install ubuntu-mainline-kernel.sh /usr/local/bin/
$ sudo ubuntu-mainline-kernel.sh -i
Finding latest version available on kernel.ubuntu.com
Latest version is: v5.7.0, continue? (y/N)
$ sudo update-grub
$ sudo grub-customizer

Since I used grub-customizer to manage Grub options, I also ran the last two commands to add the new kernel to boot options in Grub. After reboot the computer with the new kernel, I no longer have the “Dummy Output” issue any more.

Categories
Linux

Copy files from a Virtual Box managed by Vagrant

Vagrant is a very useful tool for setting up a development environment. With vagrant, you can manage Virtual Box easily. However, if you have not set up the shared directory correctly, getting access to the file in the virutal box from the local host is tricky. Below is a solution using the scp command.

vagrant ssh-config

Which gives you the identity key used by vagrant as well as the port number of the virtual box. 

  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /yourhome/.vagrant/machines/default/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

That’s all you need to put into the scp command:

scp -i /yourhome/.vagrant/machines/default/virtualbox/private_key -p 2222 -r vagrant@127.0.0.1:/vagrant/source local_dir

Notice here after “-i” is the key file reported by “vagrant ssh-config” command. Similarly the port number 2222 also came from there.

To avoid type “-i” option in the future, one can use the command below to permanently add the private key to SSH configuration so that only the port number is used in the future.

ssh-add /yourhome/.vagrant/machines/default/virtualbox/private_key

Categories
emacs

Emacs File Explorer Solution

I just came across the Treemacs package that I wish I knew earlier. Treemacs is an Emacs package that provides a tree layout file explorer for Emacs. With Treemacs, one can easily navigate through a big project tree all in Emacs.

Installation is simple through ELPA. I have the following lines in my .emacs configuration:

(setq package-archives '(("gnu" . "http://elpa.gnu.org/packages/") ))
(when (>= emacs-major-version 24)
  (require 'package)
  (add-to-list 'package-archives '("melpa" . "http://melpa.milkbox.net/packages/") t)    
  (package-initialize)
)

In Emacs, run “M-x package-refresh-contents”, then “package-install treemacs.” After installation, run “M-x treemacs” brings up the tree view.