Set up Magento 2 Development Environment with Docker & Warden

Categories Magento, Magento 2 Tags , , ,
LinkedInPinPrintShares

Warden is a CLI utility for orchestrating Docker-based developer environments and enables multiple local environments to run simultaneously without port conflicts. docker-compose is used to control everything that Warden runs.

Using Warden, a custom environment can be set up for each project. The environment can also be overridden or extended.

Warden supports the following applications and frameworks on macOS and Linux environments:

  • Magento 1
  • Magento 2
  • Laravel
  • Symfony 4
  • Shopware 6

Warden provides the following features:

Traefik: for SSL termination and routing/proxying requests into the correct containers.
Portainer: for quick visibility into what’s running inside the local Docker host.
Dnsmasq: to serve DNS responses for .test domains eliminating manual editing of /etc/hosts
SSH tunnel: for connecting from Sequel Pro or TablePlus into any one of multiple running database containers.
SSL Certificates: Warden issued wildcard SSL certificates for running https on all local development domains.

Prerequisites

Docker and docker-compose should be installed before installing Warden.

The easiest and recommended way to get Docker Compose is to install Docker Desktop.

Docker Desktop includes Docker Compose along with Docker Engine and Docker CLI which are Compose prerequisites.

Install Warden

Warden can be installed via Homebrew on both macOS and Linux hosts:


brew install davidalger/warden/warden

Start Warden Global Services


warden svc up

The started global services are run in different containers. They can be checked as below:

Check the running containers


docker ps

CONTAINER ID   IMAGE                    COMMAND                   CREATED              STATUS          PORTS                                          NAMES
42b348ff6e55   traefik:2.2              "/entrypoint.sh trae…"    About a minute ago   Up 51 seconds   127.0.0.1:80->80/tcp, 127.0.0.1:443->443/tcp   traefik
3768311d9211   panubo/sshd:1.1.0        "/entry.sh /usr/sbin…"    About a minute ago   Up 53 seconds   127.0.0.1:2222->22/tcp                         tunnel
fa246c804b67   wardenenv/mailhog:1.0    "MailHog"                 About a minute ago   Up 56 seconds   1025/tcp, 8025/tcp                             mailhog
ab1bd5f4c843   portainer/portainer-ce   "/portainer"              About a minute ago   Up 57 seconds   8000/tcp, 9000/tcp, 9443/tcp                   portainer
caaf32a6a2b2   jpillora/dnsmasq         "/bin/sh -c ' echo \"…"   About a minute ago   Up 55 seconds   127.0.0.1:53->53/udp                           dnsmasq

The above global services can be accessed from the following URLs:

https://traefik.warden.test/
https://portainer.warden.test/
https://dnsmasq.warden.test/
https://mailhog.warden.test/

Install Magento 2 using Docker & Warden

Create a new directory for your Magento 2 project and go inside that directory.

I have created a directory named your-project in the example below:


mkdir -p ~/your-path/your-project
cd ~/your-path/your-project

Generate environment file

The environment file is needed for Warden and docker to work with the project.

From your Magento 2 project’s root folder, run the following commands:


warden env-init your-project magento2

This will generate an environment file .env.

~/your-path/your-project/.env


WARDEN_ENV_NAME=your-project
WARDEN_ENV_TYPE=magento2
WARDEN_WEB_ROOT=/

TRAEFIK_DOMAIN=your-project.test
TRAEFIK_SUBDOMAIN=app

WARDEN_DB=1
WARDEN_ELASTICSEARCH=1
WARDEN_VARNISH=1
WARDEN_RABBITMQ=1
WARDEN_REDIS=1

ELASTICSEARCH_VERSION=7.6
MARIADB_VERSION=10.3
NODE_VERSION=10
COMPOSER_VERSION=1
PHP_VERSION=7.4
PHP_XDEBUG_3=1
RABBITMQ_VERSION=3.8
REDIS_VERSION=5.0
VARNISH_VERSION=6.0

WARDEN_SYNC_IGNORE=

WARDEN_ALLURE=0
WARDEN_SELENIUM=0
WARDEN_SELENIUM_DEBUG=0
WARDEN_BLACKFIRE=0
WARDEN_SPLIT_SALES=0
WARDEN_SPLIT_CHECKOUT=0
WARDEN_TEST_DB=0
WARDEN_MAGEPACK=0

BLACKFIRE_CLIENT_ID=
BLACKFIRE_CLIENT_TOKEN=
BLACKFIRE_SERVER_ID=
BLACKFIRE_SERVER_TOKEN=

Sign SSL certificate for your domain

The domain should be the same as the value of the TRAEFIK_DOMAIN variable in the above .env file.


warden sign-certificate your-project.test

Output:


==> Generating private key your-project.test.key.pem
Generating RSA private key, 2048 bit long modulus
..............+++
......................+++
e is 65537 (0x10001)
==> Generating signing req your-project.test.crt.pem
==> Generating certificate your-project.test.crt.pem
Signature ok
subject=/C=US/O=Warden.dev/CN=your-project.test
Getting CA Private Key
==> Updating traefik
Docker Compose is now in the Docker CLI, try `docker compose up`

traefik is up-to-date
Restarting traefik ... done

Start the project environment


warden env up

This will start the docker containers required for Magento 2 like php-fpm, nginx, varnish, elasticsearch, maridb, redis, and rabbitmq.

Check the running containers


docker ps

CONTAINER ID   IMAGE                                    COMMAND                   CREATED          STATUS          PORTS                                                                  NAMES
6ada603a979e   wardenenv/varnish:6.0                    "/bin/sh -c 'envsubs…"    12 minutes ago   Up 12 minutes   80/tcp                                                                 your-project_varnish_1
1a078250aa71   wardenenv/nginx:1.16                     "/bin/sh -c 'envsubs…"    12 minutes ago   Up 12 minutes   80/tcp                                                                 your-project_nginx_1
efa76857c793   wardenenv/php-fpm:7.4-magento2-xdebug3   "docker-entrypoint p…"    12 minutes ago   Up 12 minutes   9000/tcp                                                               your-project_php-debug_1
3c7e93d8b618   wardenenv/php-fpm:7.4-magento2           "docker-entrypoint p…"    12 minutes ago   Up 12 minutes   9000/tcp                                                               your-project_php-fpm_1
64a9ee90817e   wardenenv/elasticsearch:7.6              "/usr/local/bin/dock…"    12 minutes ago   Up 12 minutes   9200/tcp, 9300/tcp                                                     your-project_elasticsearch_1
ef674aa04c92   elastichq/elasticsearch-hq:latest        "supervisord -c /etc…"    12 minutes ago   Up 12 minutes   5000/tcp                                                               your-project_elasticsearch-hq_1
0f239eb38e7d   wardenenv/mariadb:10.3                   "docker-entrypoint.s…"    12 minutes ago   Up 12 minutes   3306/tcp                                                               your-project_db_1
6538d62d7740   wardenenv/rabbitmq:3.8                   "docker-entrypoint.s…"    12 minutes ago   Up 12 minutes   4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 15691-15692/tcp, 25672/tcp   your-project_rabbitmq_1
63ec5028b58b   wardenenv/redis:5.0                      "docker-entrypoint.s…"    12 minutes ago   Up 12 minutes   6379/tcp                                                               your-project_redis_1
42b348ff6e55   traefik:2.2                              "/entrypoint.sh trae…"    24 hours ago     Up 18 minutes   127.0.0.1:80->80/tcp, 127.0.0.1:443->443/tcp                           traefik
3768311d9211   panubo/sshd:1.1.0                        "/entry.sh /usr/sbin…"    24 hours ago     Up 24 hours     127.0.0.1:2222->22/tcp                                                 tunnel
fa246c804b67   wardenenv/mailhog:1.0                    "MailHog"                 24 hours ago     Up 24 hours     1025/tcp, 8025/tcp                                                     mailhog
ab1bd5f4c843   portainer/portainer-ce                   "/portainer"              24 hours ago     Up 24 hours     8000/tcp, 9000/tcp, 9443/tcp                                           portainer
caaf32a6a2b2   jpillora/dnsmasq                         "/bin/sh -c ' echo \"…"   24 hours ago     Up 24 hours     127.0.0.1:53->53/udp                                                   dnsmasq

Configure global Magento Marketplace credentials


composer global config http-basic.repo.magento.com <username> <password>

The username and password are from the Magento Marketplace Access Keys.

Magento Docs: Get your authentication keys

Username = Public Key
Password = Private Key

The above command will generate ~/.composer/auth.json file, e.g.


{
    "http-basic": {
        "repo.magento.com": {
            "username": "your-public-key-xxxxxxxx",
            "password": "your-private-key-xxxxxxx"
        }
    }
}

If you already have the ~/.composer/auth.json file in your host machine, then you don’t need to run the above command to generate the global composer file. The auth.json file is mounted into the docker container.

Go inside the php-fpm container shell


warden shell

Output:


www-data@your-project-php-fpm:/var/www/html$

It’s the same as running the following command:


docker exec -it your-project_php-fpm_1 bash

Initialize project source files


META_PACKAGE=magento/project-community-edition META_VERSION=2.4.x

composer create-project --repository-url=https://repo.magento.com/ \
"${META_PACKAGE}" /tmp/your-project "${META_VERSION}"

rsync -a /tmp/your-project/ /var/www/html/
rm -rf /tmp/your-project/

Check project files


www-data@your-project-php-fpm:/var/www/html$ ls -al

total 1804
drwxrwxr-x 12 www-data www-data   4096 Nov 18 05:45 .
drwxr-xr-x  1 root     root       4096 Aug  1 04:34 ..
-rw-rw-r--  1 www-data www-data    343 Nov 18 05:45 .editorconfig
-rw-r--r--  1 www-data www-data    657 Nov 18 03:55 .env
-rw-rw-r--  1 www-data www-data   1571 Nov 18 05:45 .gitignore
-rw-rw-r--  1 www-data www-data    214 Nov 18 05:45 .htaccess
-rw-rw-r--  1 www-data www-data  11382 Nov 18 05:45 .htaccess.sample
-rw-rw-r--  1 www-data www-data   1523 Nov 18 05:45 .php-cs-fixer.dist.php
-rw-rw-r--  1 www-data www-data    101 Nov 18 05:45 .user.ini
-rw-rw-r--  1 www-data www-data 766543 Nov 18 05:45 CHANGELOG.md
-rw-rw-r--  1 www-data www-data    650 Nov 18 05:45 COPYING.txt
-rw-rw-r--  1 www-data www-data   2972 Nov 18 05:45 Gruntfile.js.sample
-rw-rw-r--  1 www-data www-data  10364 Nov 18 05:45 LICENSE.txt
-rw-rw-r--  1 www-data www-data  10376 Nov 18 05:45 LICENSE_AFL.txt
-rw-rw-r--  1 www-data www-data    698 Nov 18 05:45 SECURITY.md
drwxr-xr-x  4 www-data www-data   4096 Nov 18 05:45 app
-rw-rw-r--  1 www-data www-data    150 Nov 18 05:45 auth.json.sample
drwxr-xr-x  2 www-data www-data   4096 Nov 18 05:45 bin
-rw-r--r--  1 www-data www-data   2800 Nov 18 05:35 composer.json
-rw-rw-r--  1 www-data www-data 927187 Nov 18 05:45 composer.lock
drwxr-xr-x  4 www-data www-data   4096 Nov 18 05:45 dev
drwxr-xr-x  2 www-data www-data   4096 Nov 18 05:45 generated
-rw-rw-r--  1 www-data www-data     55 Nov 18 05:45 grunt-config.json.sample
drwxr-xr-x  4 www-data www-data   4096 Nov 18 05:45 lib
-rw-rw-r--  1 www-data www-data   6957 Nov 18 05:45 nginx.conf.sample
-rw-rw-r--  1 www-data www-data   1234 Nov 18 05:45 package.json.sample
drwxr-xr-x  2 www-data www-data   4096 Nov 18 05:45 phpserver
drwxr-xr-x  6 www-data www-data   4096 Nov 18 05:45 pub
drwxr-xr-x  7 www-data www-data   4096 Nov 18 05:45 setup
drwxr-xr-x  2 www-data www-data   4096 Nov 18 05:45 var
drwxrwxr-x 75 www-data www-data   4096 Nov 18 05:45 vendor

Install Magento2 Application


bin/magento setup:install \
    --backend-frontname=backend \
    --amqp-host=rabbitmq \
    --amqp-port=5672 \
    --amqp-user=guest \
    --amqp-password=guest \
    --db-host=db \
    --db-name=magento \
    --db-user=magento \
    --db-password=magento \
    --search-engine=elasticsearch7 \
    --elasticsearch-host=elasticsearch \
    --elasticsearch-port=9200 \
    --elasticsearch-index-prefix=magento2 \
    --elasticsearch-enable-auth=0 \
    --elasticsearch-timeout=15 \
    --http-cache-hosts=varnish:80 \
    --session-save=redis \
    --session-save-redis-host=redis \
    --session-save-redis-port=6379 \
    --session-save-redis-db=2 \
    --session-save-redis-max-concurrency=20 \
    --cache-backend=redis \
    --cache-backend-redis-server=redis \
    --cache-backend-redis-db=0 \
    --cache-backend-redis-port=6379 \
    --page-cache=redis \
    --page-cache-redis-server=redis \
    --page-cache-redis-db=1 \
    --page-cache-redis-port=6379

Output:


...
...
[SUCCESS]: Magento installation complete.
[SUCCESS]: Magento Admin URI: /backend
Nothing to import.

Configure Magento2 Application


bin/magento config:set --lock-env web/unsecure/base_url \
    "https://${TRAEFIK_SUBDOMAIN}.${TRAEFIK_DOMAIN}/"

bin/magento config:set --lock-env web/secure/base_url \
    "https://${TRAEFIK_SUBDOMAIN}.${TRAEFIK_DOMAIN}/"

bin/magento config:set --lock-env web/secure/offloader_header X-Forwarded-Proto

bin/magento config:set --lock-env web/secure/use_in_frontend 1
bin/magento config:set --lock-env web/secure/use_in_adminhtml 1
bin/magento config:set --lock-env web/seo/use_rewrites 1

bin/magento config:set --lock-env system/full_page_cache/caching_application 2
bin/magento config:set --lock-env system/full_page_cache/ttl 604800

bin/magento config:set --lock-env catalog/search/enable_eav_indexer 1

bin/magento config:set --lock-env dev/static/sign 0

bin/magento deploy:mode:set -s developer
bin/magento cache:disable block_html full_page

bin/magento indexer:reindex
bin/magento cache:flush

Search-engine & Elasticsearch Configuration for Magento 2.4.x and above

  • Only for Magento 2.4.x and above.
  • Not required/supported for Magento 2.3.x and below.

bin/magento config:set --lock-env catalog/search/engine elasticsearch7
bin/magento config:set --lock-env catalog/search/elasticsearch7_server_hostname elasticsearch
bin/magento config:set --lock-env catalog/search/elasticsearch7_server_port 9200
bin/magento config:set --lock-env catalog/search/elasticsearch7_index_prefix magento2
bin/magento config:set --lock-env catalog/search/elasticsearch7_enable_auth 0
bin/magento config:set --lock-env catalog/search/elasticsearch7_server_timeout 15

Generate Magento Admin User

The admin username is set as localadmin below:


ADMIN_PASS="$(pwgen -n1 16)"
ADMIN_USER=localadmin

bin/magento admin:user:create \
    --admin-password="${ADMIN_PASS}" \
    --admin-user="${ADMIN_USER}" \
    --admin-firstname="Local" \
    --admin-lastname="Admin" \
    --admin-email="${ADMIN_USER}@example.com"
printf "u: %s\np: %s\n" "${ADMIN_USER}" "${ADMIN_PASS}"

Launch the application

Frontend: https://app.your-project.test/
Backend: https://app.your-project.test/backend/
Rabbitmq: https://rabbitmq.your-project.test/
Elasticsearch: https://elasticsearch.your-project.test/

Global Services

https://traefik.warden.test/
https://portainer.warden.test/
https://dnsmasq.warden.test/
https://mailhog.warden.test/

Common Warden Commands


  svc               Orchestrates global services such as traefik, portainer and dnsmasq via docker-compose
  env-init          Configure environment by adding '.env' file to the current working directory
  env               Controls an environment from any point within the root project directory
  db                Interacts with the db service on an environment (see 'warden db -h' for details)
  redis             Interacts with the redis service on an environment (see 'warden redis -h' for details)
  install           Initializes or updates warden configuration on host machine
  shell             Launches into a shell within the current project environment
  debug             Launches debug enabled shell within current project environment
  sign-certificate  Signs a wildcard certificate including all passed hostnames on the SAN list
  version           Show version information

Warden usage:


# Launch shell session inside php-fpm container
warden shell

# Create and start containers
warden env up

# Stop and remove containers
warden env down

# Remove volumes completely
warden env down -v

# Start services/containers
warden env start

# Stop services/containers
warden env stop

# Connect to redis
warden redis

# Flush redis completely
warden redis flushall

# Run redis continous stat mode
warden redis --stat

# Tail the varnish activity log
warden env exec -T varnish varnishlog

# Flush varnish
warden env exec -T varnish varnishadm 'ban req.url ~ .' 

# Tail environment nginx and php logs
warden env logs --tail 0 -f nginx php-fpm php-debug

# Monitor database processlist
watch -n 3 "warden db connect -A -e 'show processlist'"

# Import a database (if you don’t have pv installed, use cat instead)
pv /path/to/dump.sql.gz | gunzip -c | warden db import

Hope this helps. Thanks.

Related posts:

  1. Set up PHP Development Environment with Docker
  2. Magento 2: Useful Configuration Settings & CLI Commands for Development Environment
  3. Introduction to Docker and its Tools & Terminologies
  4. Magento 2: Increase Admin Login Session Lifetime
  5. Magento 2: Creating & Running Cron in Custom Module