Club Drupal

Afin de gagner du temps et d'éviter les erreurs humaines, des scripts ont été écrits tout au long du projet. Certains sont même exécutés automatiquement toutes les semaines.

{{>toc}}

h1. Les tâches planifiées

Pour exécuter ces tâches régulièrement, on utilise le "crontab":http://fr.wikipedia.org/wiki/Crontab. Il s'agit d'un programme installé sur notre serveur.

Pour voir et modifier la liste des actions :

# se connecter au serveur : <code>ssh assos@sas1.centrale-marseille</code>

# taper la commande pour voir le crontab <code>crontab -l</code>

# taper la commande pour modifier le crontab <code>crontab -e</code>. /!\ Ne pas effectuer cette opération avant de s'être renseigné sur "vi":http://fr.wikipedia.org/wiki/Vi (l'éditeur de texte utilisé qui n'est pas vraiment intuitif :p) et sur "la syntaxe du crontab":http://fr.wikipedia.org/wiki/Crontab#Modification !

{{tip(Pour utiliser un autre editeur que vi taper : export EDITOR=editor_name exemple : vim, nano)}}

Voici la liste des tâches effectuées régulièrement.

h2. La réinitialisation des variables

Cette tâche consiste à réinitialiser certaines variables qui donnent des droits considérés comme trop permissifs donc dangereux aux administrateurs des sites ou nécessaire pour d'autres scripts tels que piwik ou security_review.

En voici quelques exemples :

<pre>

<code class="bash">

drush -y vset error_level 0

</code>

</pre>

Cette commande permet de ne pas afficher les messages d'erreurs. En effet, ils contiennent parfois des informations sensibles sur l'installation et ne doivent donc pas être divulgués à n'importe qui.

<pre>

<code class="php">

drush -y vset --always-set reverse_proxy TRUE

drush -y vset --always-set --format=json reverse_proxy_addresses '["147.94.19.16","147.94.19.17"]'

</code>

</pre>

Ces commandes permettent de déclarer à Drupal les serveurs proxy du CRI afin d'éviter qu'il ne répertorie tous les visiteurs comme ayant l'adresse des serveurs sus-cités. **Pas fini : voir http://assos.centrale-marseille.fr/content/t%C3%A2che/d%C3%A9clarer-les-proxy-du-cri-%C3%A0-drupal**

h2. La sauvegarde des bases de données

Sur les installations d6 et d7 : une fois par semaine

On utilise les [[Scripts_et_taches_planifiees#Tout-sauvegarder|scripts de sauvegarde créés par le club Drupal]].

h2. La purge des sauvegardes de bdd

Sur les installations d6 et d7 : toutes les semaines

Cette tâche utilise le script de [[Scripts_et_taches_planifiees#purge_des_sauvegardes.sh|purge des sauvegardes]] afin de libérer de l'espace disque en supprimant les sauvegardes de bdd les plus vieilles.

h1. Liste des scripts à disposition

Les scripts utilisés sont hébergés dans le répertoire bin du compte assos.

h2. all-reset-permissions.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

# This script puts the correct permissions to sites folders, settings.php and scripts.

######### drupal 6

for dir in `find $d6_dir_sites -type d -maxdepth 1 -mindepth 1 ! -name all ! -name images ! -name languages` ; do

    chmod 755 $dir

    chmod 400 $dir/settings.php

done

######### drupal 7

for dir in `find $d7_dir_sites -type d -maxdepth 1 -mindepth 1 ! -name all` ; do

    chmod 755 $dir

    chmod 400 $dir/settings.php

done

####### bin

chmod -R 700 $dir_scripts

</code>

</pre>

h2. contrib-usage.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

. scripts-utils.sh

# ARGS: Drupal_version, project_status, project_name

# List sites that have the project_name with the corresponding project_status.

check_arguments $# 3

if [ $1 = d6 ] ; then

    cd $d6_dir_sites

else

    cd $d7_dir_sites

fi

for dir in `find . -maxdepth 1 -mindepth 1 -type d ! -name all ! -name languages ! -name images` ; do

    # List non-core projects that correspond to the status.

    # Keep project_name if listed.

    # Count line result. 0 if not listed or 1 if listed.

	# Print site_dir if listed.

    cd $dir;

	if [ 1 -eq `drush pml --no-core --status=$2 | grep $3 | wc -l` ] ; then

        echo $dir; 

    fi

done

</code>

</pre>

h2. d6-all-drush.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

. scripts-utils.sh

# ARGS: drush_command

check_arguments $# 1

for dir in `find $d6_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all ! -name languages ! -name images` ; do

    cd $dir

    echo $dir

    drush $*

done

</code>

</pre>

h2. d6-all-dump-full.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"`

# Dump D6 database with webassos user.

mysqldump --single-transaction webassos -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY > $d6_dir_full_backup/webassos/$current_date.webassos.sql

# Dump D6 form database with forum user.

mysqldump forum -h myweb.serv.int -u forum --password=dtcAltF12 > $d6_dir_full_backup/forum/$current_date.forum.sql

</code>

</pre>

h2. d6-all-update-contrib.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

d6-all-drush.sh -y en update

# Launch cron to check for updates.

d6-all-drush.sh -q cron

d6-all-drush.sh cc all

d6-all-dump-full.sh

d6-all-drush.sh -y upc --no-core

d6-all-drush.sh -y updb

d6-all-drush.sh -y dis update

# Cron report.

d6-all-drush.sh cron

</code>

</pre>

h2. d6-dump-individual.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

# ARGS: auto or manual, site_prefix

# WARNING : backup are regularly flushed, put your backup in a safe place.

cd $dir_individual_backup

#récupération des tables du site dans le fichier liste_tables.temp

tables='_%'

liste="$1$tables"

mysql -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY -BNe "show tables like '"$liste"'" webassos | tr '\r\n' ' ' > liste_tables.temp

#transformation de cette liste en une variable

var=$(cat liste_tables.temp)

#sauvegarde de toutes ces tables

current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"`

suffixe="_dump$current_date.sql"

fichier="$1$suffixe"

if [ $1 = 'auto' ] ; then

        mysqldump webassos -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY $var > $d6_dir_individual_auto_backup/$fichier

    else

        mysqldump webassos -h myweb.serv.int -u webassos --password=HBVH2ljgyZCA0AP251DY $var > $d6_dir_individual_manual_backup/$fichier

    fi

#suppression du fichier temporaire utilisé

rm liste_tables.temp

</code>

</pre>

h2. d6-reset-variables.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

cd $d6_dir_sites

d6-all-drush.sh vset --always-set reverse_proxy TRUE

d6-all-drush.sh vset --always-set --format=json reverse_proxy_addresses '["147.94.19.16","147.94.19.17"]'

drush @sites ev "variable_set('update_notify_emails', array($email_multi_assos));" --yes

</code>

</pre>

h2. d7-all-drush-logged.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

. scripts-utils.sh

# ARGS: drush_command

check_arguments $# 1

for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do

    cd $dir

    current_date=`date "+%Y-%m-%d_%Hh:%Mm"`

    echo "======$current_date======" >> $d7_dir_log/$x.log

    echo "Arguments are: $* " >> $d7_dir_log/$x.log

    drush $* >> $d7_dir_log/$x.log

    #To have the same caracters limit we needed 'end' that have 4 caracters, we used the Croate language for this ! LOL 

    echo -e "=============KRAJ=============\n" >> $d7_dir_log/$x.log

done

</code>

</pre>

h2. d7-all-drush.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

. scripts-utils.sh

# ARGS: drush_command

check_arguments $# 1

for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do

    cd $dir

    echo $dir

    drush $*

done

</code>

</pre>

h2. d7-all-dump-full.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"`

# Dump D7 databases at once.

mysqldump -h $db_server -u $db_user -pNoNo82jJ --all-databases > $d7_dir_full_backup/myassos/$current_date.d7_full.sql

</code>

</pre>

h2. d7-all-dump-individual.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

# ARGS: auto or manual

current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"`

for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do

    cd $dir

    drush cc all

    if [ $1 = 'auto' ] ; then

        drush sql-dump --result-file="$d7_dir_individual_auto_backup/$dir/$current_date.$dir.sql"

    else

        drush sql-dump --result-file="$d7_dir_individual_manual_backup/$dir/$current_date.$dir.sql"

    fi

done

</code>

</pre>

h2. d7-all-releases-info.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

# List version of a module or theme that exists in sites/all/*

# and all versions that are available for same project on drupal.org

# modules

for dir in `find $d7_dir_sites/all/modules -maxdepth 1 -mindepth 1 -type d` ; do

    drush pm-releases $dir

done

# themes

for dir in `find $d7_dir_sites/all/themes -maxdepth 1 -mindepth 1 -type d` ; do

    drush pm-releases $dir

done

</code>

</pre>

h2. d7-all-reset-variables.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

for dir in `find $d7_dir_sites -maxdepth 1 -mindepth 1 -type d ! -name all` ; do

    cd $dir

    echo $dir

    d7-reset-variables.sh

done

</code>

</pre>

h2. d7-all-security-review.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

# grep -v success to keep site_name and error lines.

bash -c 'd7-all-drush.sh security-review |& grep -v success | mail -s "Security review" $email_multi_assos'

</code>

</pre>

h2. d7-all-update-contrib.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

d7-all-drush.sh -y en update

# Launch cron to check for updates.

d7-all-drush.sh -q cron

d7-all-drush.sh cc all

d7-all-dump-individual.sh

d7-all-drush-logged.sh -y upc --no-core

d7-all-drush.sh -y updb

d7-all-drush.sh -y dis update

# Cron report.

d7-all-drush.sh cron | mail -s "Cron report: d7-all-update-contrib.sh" $email_multi_assos

</code>

</pre>

h2. d7-all-update-localisation.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

d7-all-drush.sh -y en l10n_update

d7-all-drush.sh l10n-update-refresh

d7-all-drush.sh l10n-update

d7-all-drush.sh -y dis l10n_update

</code>

</pre>

h2. d7-create-site.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

. scripts-config-site.sh $1

. scripts-utils.sh

# ARGS: site name

######## Exceptions

check_arguments $# 1

# "-" is forbidden because it provokes database error.

if `echo $1 | grep -` ; then

    echo '"-" is forbidden in the site name'

    exit 1

fi

# Site name length must be lower or equal to 16 due to database limitations.

if [ $(echo $1 | wc -n) -le 16 ] ; then

    echo "site name can't have more than 16 characters"

    exit 1

fi

###### Initialisation

cd $d7_dir

db_password=`ask_password_db $db_server $db_user`

site_password=`gen_password`

# Check if site database already exists.

if mysql -h $db_server -u $db_user -e "USE $d7_site_name" -p$db_password 2>/dev/null ; then

    echo "Database $d7_site_name already exists"

    exit 1

fi

# Check if site folder already exists.

if [ -d $d7_site_dir ] ; then

    echo "Folder $d7_site_dir already exists"

    exit 1

fi

###### Main

mkdir $d7_site_dir

# Backup requirements

mkdir $d7_dir_individual_auto_backup/$d7_site_name

mkdir $d7_dir_individual_manual_backup/$d7_site_name

current_date=`date "+%Y-%m-%d-%Hh%Mm%Ss"`

touch $d7_dir_individual_auto_backup/$d7_site_name/$current_date.sql

touch $d7_dir_individual_auto_backup/$d7_site_name/$current_date.sql

touch $d7_dir_individual_auto_backup/$d7_site_name/$current_date.sql

# Create and grant privileges on database

mysql -h $db_server -u $db_user -e "CREATE DATABASE $d7_site_name" -p$db_password

mysql -h $db_server -u $db_user -e "GRANT ALL PRIVILEGES ON $d7_site_name.* TO '$d7_site_name'@'%' IDENTIFIED BY '$site_password'" -p$db_password

# Create settings.php

sed "s/\%\%DBUSER\%\%/$d7_site_name/ ; s/\%\%DBNAME\%\%/$d7_site_name/ ; s/\%\%DBPASS\%\%/$site_password/ ; s/\%\%nomsite\%\%/$d7_site_name/" < $d7_settings_template > $d7_site_settings

# Create symbolic link

cd $d7_dir

ln -s . $d7_site_name

# Next Instructions

echo "Go to http://assos.centrale-marseille.fr/$d7_site_name/install.php to continue."

echo "Press enter when ready to go on."

read key

# Init variables

d7-reset-variables.sh $d7_site_name

# Permissions

chmod -R 755 $d7_site_dir

chmod 400 $d7_site_settings

# Last instructions

echo "Last instructions:"

echo "- Advice the webmaster to close account creation on the website"

echo "- Give the webmaster a link to the club Drupal's tutorials "

echo "- Create a node of type \"Site\" on default"

echo "- Register the webmaster on webmasters@listes.centrale-marseille.fr"

</code>

</pre>

h2. d7-delete-site.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

. scripts-config-site.sh $1

. scripts-utils.sh

# ARGS: site name.

check_arguments $# 1

db_password=`ask_password_db $db_server $db_user`

# Delete database.

mysql -h $db_server -u $db_user -p$db_password -e "IF EXISTS DROP DATABASE $d7_site_name"

# Delete MYSQL user.

mysql -h $db_server -u $db_user -p$db_password -e "DROP USER '$d7_site_name'@'%'"

# Delete site's folder.

chmod -R 700 $d7_site_dir

rm -r $d7_site_dir

# Delete symbolic link.

rm $d7_dir/$d7_site_name

# Delete database backups.

rm -r $d7_dir_individual_auto_backup/assos.centrale-marseille.fr.$d7_site_name

rm -r $d7_dir_individual_manual_backup/assos.centrale-marseille.fr.$d7_site_name

echo "Don't forget to:"

echo "- Refresh node site on default"

echo "- Check particular behavior (normally detailed on site node)"

</code>

</pre>

h2. d7-reset-variables.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

# ARGS: [site_name]

if [ ! -z $1 ] ; then

    . scripts-config-site.sh $1

    cd $d7_site_dir

fi

current_timestamp=`date "+%s"`

####### General variables

drush -y vset --always-set reverse_proxy TRUE

drush -y vset --always-set --format=json reverse_proxy_addresses '["147.94.19.16","147.94.19.17"]'

drush -y ev "variable_set('update_notify_emails', array($email_multi_assos));"

drush -y vset error_level 0

###### Hidden captcha

drush -y en hidden_captcha

# Log wrong answers.

drush -y vset captcha_log_wrong_responses 1

# Use hidden captcha for all forms.

drush -y sqlq --db-prefix "UPDATE {captcha_points} SET module = 'hidden_captcha', captcha_type = 'Hidden CAPTCHA' WHERE module is NULL;"

# Flush captcha cache.

drush -y vdel captcha_placement_map_cache

# Randomely generate a math question as the label of the hidden captcha field.

drush -y vset hidden_captcha_label "$RANDOM + $RANDOM"

####### Piwik

drush -y en piwik

drush -y vset piwik_site_id "101"

drush -y vset piwik_url_http "http://piwik.centrale-marseille.fr/"

drush -y vset piwik_url_https "https://piwik.centrale-marseille.fr/"

# Piwik cache.

drush -y vset piwik_cache 1

drush -y vset piwik_visibility_roles "1"

# Anonymous and authenticated statistics.

drush -y vset --format=json piwik_roles '{"1":0,"2":0}'

drush -y vset piwik_page_title_hierarchy 1

# Activate local search.

drush -y vset piwik_site_search 1

###### Security review

# For untrusted roles:

# 1 : anonymous user

# 2 : authenticated user

# 3 : administrator

drush -y en security_review

drush -y ev "variable_set('security_review_untrusted_roles', array('1'));"

# Security review can't check for files permissions on multi_assos.

drush -y sqlq --db-prefix "UPDATE {security_review} SET skip = '1', skiptime = $current_timestamp, skipuid= '1' WHERE reviewcheck IN ('file_perms');"

drush -y sqlq --db-prefix "UPDATE {security_review} SET skip = '0', skiptime = 0, skipuid= NULL WHERE reviewcheck NOT IN ('file_perms');"

</code>

</pre>

h2. delete-temporary-files.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

find $dir_multi_assos -name '*~' -delete

</code>

</pre>

h2. disk-usage.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

echo -e "Quotas: 9.1 GB\n"

echo "Assos:"

du -hcs $dir_multi_assos | grep total

echo -e "\n"

echo "D6:"

cd $d6_dir_sites

du -h -d 1

echo -e "\n"

echo "D7:"

cd $d7_dir_sites

du -h -d 1

echo -e "\n"

echo "Logs:"

du -hcs $dir_log | grep total

echo -e "\n"

echo "Backups:"

cd $dir_backup

du -h -d 2

</code>

</pre>

h2. flush-files.sh

<pre>

<code class="php">

#!/bin/sh

# ARGS: number of files to keep, email address

# usage: flush files from a folder except a number equal to argument.

# Check if scripts-config.sh is imported.

if [ -z $scripts_utils ] ; then

    echo "Import of scripts-utils.sh required."

    exit 1

fi

check_arguments $# 2

backups_number=`ls | wc -l`

let "number_of_backups_to_delete = backups_number - $1"

if [ $number_of_backups_to_delete -gt 0 ] ; then

    ls | head -$number_of_backups_to_delete | xargs rm

else

    dir=`pwd`

    echo "There are not enough files in $dir to Flush it. Check if backup script works fine." | mail -s "[db] $dir has a backup problem" $2

fi

</code>

</pre>

h2. flush-full-databases-backups.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

# ARGS: Drupal version

cd $dir_full_backup/$1

for dir in `ls` ; do

    cd $dir

    flush-files.sh $db_full_backup_number $email_multi_assos

done

</code>

</pre>

h2. flush-individual-databases-backups.sh

<pre>

<code class="php">

#!/bin/sh

. scripts-config.sh

# ARGS: manual or auto, Drupal version

cd $dir_individual_backup/$1/$2

for dir in `ls` ; do

    cd $dir

    if [ $1 = 'auto' ] ; then

        flush-files.sh $db_individual_auto_backup_number $email_multi_assos

    else

        flush-files.sh $db_individual_manual_backup_number $email_multi_assos

    fi

done

</code>

</pre>

h2. scripts-config-site.sh

<pre>

<code class="php">

#!/bin/sh

# Sets useful variables of a site.

# ARGS: site_name

# Requires scripts-config.sh

# Check if scripts-config.sh is imported.

if [ -z $scripts_config ] ; then

    echo "Import of scripts-config.sh required."

    exit 1

fi

if [ -z $1 ] ; then

    echo "This script needs a site name as argument."

    exit 1

fi

scripts_config_site='imported'

d7_site_name=$1

d7_site_dir="$d7_dir_sites/assos.centrale-marseille.fr.$d7_site_name"

d7_site_settings="$d7_site_dir/settings.php"

</code>

</pre>

h2. scripts-config.sh

<pre>

<code class="php">

#!/bin/sh

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/users/guest/assos/bin

PATH=$PATH:/usr/guest/assos/bin

scripts_config='imported'

email_multi_assos="assos@centrale-marseille.fr"

dir_multi_assos='/users/guest/assos'

dir_scripts="$dir_multi_assos/bin"

dir_template="$dir_multi_assos/Desktop"

###### Database

db_server="myassos.serv.int"

db_user="assos"

dir_log="$dir_multi_assos/log"

###### Backup

db_full_backup_number=3

db_individual_manual_backup_number=3

db_individual_auto_backup_number=3

dir_backup="$dir_multi_assos/backup"

dir_full_backup="$dir_backup/full"

dir_individual_backup="$dir_backup/individual"

dir_individual_manual_backup="$dir_individual_backup/manual"

dir_individual_auto_backup="$dir_individual_backup/auto"

###### D7 variables

d7_dir="$dir_multi_assos/htmltest"

d7_dir_sites="$d7_dir/sites"

### Log

d7_dir_log="$dir_log/d7"

### Backup

d7_dir_full_backup="$dir_full_backup/d7"

d7_dir_individual_manual_backup="$dir_individual_manual_backup/d7"

d7_dir_individual_auto_backup="$dir_individual_auto_backup/d7"

### Template

d7_settings_template_name="settings-D7-bddinde-template.php"

d7_settings_template="$dir_template/$d7_settings_template_name"

###### D6 variables

d6_dir="$dir_multi_assos/html"

d6_dir_sites="$d6_dir/sites"

### Backup

d6_dir_full_backup="$dir_full_backup/d6"

d6_dir_individual_manual_backup="$dir_individual_manual_backup/d6"

d6_dir_individual_auto_backup="$dir_individual_auto_backup/d6"

</code>

</pre>

h2. scripts-utils.sh

<pre>

<code class="php">

#!/bin/sh

# This script contains useful functions for other scripts.

# Check if scripts-config.sh is imported.

if [ -z $scripts_config ] ; then

    echo "Import of scripts-config.sh required."

    exit 1

fi

scripts_utils='imported'

ask_password() {

    # read -s doesn't work with sh.

    # usage: pass=`ask_password "password please:"`

    echo $1 >&2

    echo -n ">" >&2

    stty_avant=`stty -g`

    stty -echo

    read p

    stty $stty_avant

    echo "$p"

    unset p

}

ask_password_db() {

    # ARGS: server_name, user_name

    local db_password="pour_boucler"

    # empty db request to validate password

    while ! mysql -h $1 -u $2 -p$db_password -e "" 2>/dev/null ; do

        db_password=`ask_password "database password:"`

        echo ""

    done

}

generate_password(){

    # ARGS: [password_length]

    # The password contains special characters. '/' must be excluded to avoid sed malfunction.

    local site_password='/'

    if [ -z $1 ] ; then

        local password_length=20

    else

        local password_length=$1

    fi

    while echo "$site_password" | grep -Fq '/' ; do

        site_password=`dd if=/dev/urandom count=1 | uuencode -m - | head -n 2 | tail -n 1 | cut -c-$password_length`

    done

    echo $site_password

}

count_d7_sites(){

    find $d7_dir_sites -type d ! -name all -maxdepth 1 | wc -l

}

check_arguments() {

    # ARGS: number of arguments passed to script, number of arguments required

    if [ $1 -le $2  ] ; then

        echo "Number of arguments insuffisant."

        exit 1

    fi

}

</code>

</pre>