Index by title

Updated over 6 years ago by Jessie Lee

101 Uses for cURL

  1. Query a website curl http(s)://web.site
  2. Query a websites info (http flags etc.) curl -I http(s)://web.site # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
  3. # # # # # # # # # # # # # # # # # # # # # # #

ADAP Hosting Plan changelog

Version 2

Version 1


Add another user's mailbox in Outlook 2007

Permanent use

One-time use


Amazon SES for CiviCRM

Pre-requisites: DNS access

You now have two options for SMTP setup:

SMTP direct from CiviCRM

SMTP relayed through Postfix (recommended)

See also here: http://docs.aws.amazon.com/ses/latest/DeveloperGuide/postfix.html

Postfix will give higher performance on large mailings, and also does a much better job of handling resends, greylisting, etc. This should be implemented wherever possible.

Bounce processing setup

There's code here to use a Amazon SNS to a bounce processing address instead of the typical bounce processing here. I have NOT tested this, but I don't think this provides an advantage over typical bounce processing. An SNS endpoint in CiviCRM COULD provide higher performance than IMAP/POP-based bounce processing, I don't think this solution is it.

So instead, just see Set up local bounce processing.

Testing

Troubleshooting

In testing if CiviCRM reports that everything's been sent correctly, but then no emails are received, the problem could be with Postfix authenticating to SES. /var/log/mail.log might contain entries like this.

Aug  9 11:37:48 XXXXXXXX postfix/smtpd[24665]: connect from localhost[127.0.0.1]
Aug  9 11:37:48 XXXXXXXX postfix/smtpd[24665]: A30A4624F1: client=localhost[127.0.0.1]
Aug  9 11:37:48 XXXXXXXX postfix/cleanup[24669]: A30A4624F1: message-id=<20160809163748.A30A4624F1@XXXXXXXX.example.org>
Aug  9 11:37:48 XXXXXXXX postfix/qmgr[24664]: A30A4624F1: from=<XXXXXXXX@example.org>, size=453, nrcpt=1 (queue active)
Aug  9 11:37:48 XXXXXXXX postfix/smtpd[24665]: disconnect from localhost[127.0.0.1]
Aug  9 11:37:48 XXXXXXXX postfix/smtp[24670]: warning: SASL authentication failure: No worthy mechs found
Aug  9 11:37:48 XXXXXXXX postfix/smtp[24670]: A30A4624F1: SASL authentication failed; cannot authenticate to server email-smtp.us-west-2.amazonaws.com[54.68.106.242]: no mechanism available
Aug  9 11:37:48 XXXXXXXX postfix/smtp[24670]: warning: SASL authentication failure: No worthy mechs found
Aug  9 11:37:48 XXXXXXXX postfix/smtp[24670]: A30A4624F1: SASL authentication failed; cannot authenticate to server email-smtp.us-west-2.amazonaws.com[52.35.58.187]: no mechanism available
Aug  9 11:37:49 XXXXXXXX postfix/smtp[24670]: warning: SASL authentication failure: No worthy mechs found
Aug  9 11:37:49 XXXXXXXX postfix/smtp[24670]: A30A4624F1: to=<XXXXXXXX@example.com>, relay=email-smtp.us-west-2.amazonaws.com[52.35.228.26]:25, delay=0.39, delays=0.04/0.01/0.34/0, dsn=4.7.0, status=deferred (SASL authentication failed; cannot authenticate to server email-smtp.us-west-2.amazonaws.com[52.35.228.26]: no mechanism available)

If that's the case, you might need to install some additional libraries, namely the libsasl2-modules package in Debian/Ubuntu systems.

SPECIAL NOTE FOR EC2 INSTANCES

EC2 severely throttles outgoing mail on ports 25/465/587. You need to submit a request to Amazon to have this throttle lifted. This is separate from any user-configurable firewalling! You can do that (and set up PTR) here: https://portal.aws.amazon.com/gp/aws/html-forms-controller/contactus/ec2-email-limit-rdns-request


Basic Mapping Documentation

Modules to be installed

OpenLayers javascript libraries installed in sites/all/libraries.

The data format for mapping with OpenLayers in Drupal is handled by content types with particular data type fields associated with those content types. We've covered two field data types, which allow us to map points, more specifically longitude and latitude. Those fields are Postal Address field and Geofield.

If we use postal address field, we also have to use geofield. First add the postal address field; there's only a single widget option, Dynamic address form. In the configuration, set this field to be required, filter to particular countries if that's needed for data entry. The rest of the defaults should be fine. Then add a geofield to the content type in order to transform the entered address into a longitude and latitude. The widget for this field needs to be set to Geocode from another field. Set this field to be required, choose the address field as the Geocode from field that will be transformed into geolocation data, and finally choose the geocoding service. We used Google during the demo, but there are privacy issues to be discussed around actual data. When entering data, only the address field appears on the edit form, but when viewing the entered content the longitude and latitude appear as well.

If we use just the geofield, there are several options. We've covered the map-based data entry option. When adding the geofield to the content type, select Openlayers Map as the widget. Set the field to be required and adjust the default map center and zoom for data entry if that's needed. The rest of the defaults should be fine to accept. When editing the content, the map appears for the user to enter the data, but only the longitude and latitude appear when viewing it.

The geofield supports other types of data, such as direct input of longitude and latitude and two dimensional shapes.

With some data now entered as content in Drupal, we need to create a map. Creating a map with OpenLayers in Drupal requires three components
  1. Aggregating the data and transforming from the Drupal format to standardized mapping format
  2. Creating the map that assembles base and overly layers
  3. Creating the page where the map is rendered
  1. Mapping with OpenLayers in Drupal uses Views to aggregate the data.
  2. The OpenLayers modules provides several example maps that we're going to clone and configure for the map we want to create. Under Structure > OpenLayers > Maps, there should be a list of the example maps (and any other maps we've created in the past). Click the down arrow in the Operations column for the map we want to clone and click the Clone option.
  3. OpenLayers with Drupal also uses Views to render the map we just created in Step 2 onto a page that renders the enabled base and overlay layers into a single map that the user can interact with. This can be done with either a completely separate view or by adding a Page display to the data aggregation view we created in Step 1.

The map should now be rendered at the path given in the last step.


Borg backups

This is documentation and scripts for using Borg backups with backupninja. This relies heavily on the work of Jon Goldberg, Benzhaomin, and Guillaume Subiron.

What is Borg?

From the Borg docs:

BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption.

The main goal of Borg is to provide an efficient and secure way to backup data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to not fully trusted targets.

Usage

Main borg docs: https://borgbackup.readthedocs.io/en/stable/

List backups

borg list /path/to/repo

Mount a single backup

borg mount /path/to/repo::individual_backup /tmp/mymountpoint

Check viability of the repo

borg check -v /path/to/repo

For help fixing a repo that fails borg check see this example from work on VCW osTicket.

If the backup has a passphrase, you'll need to enter it to do any of those commands. The passphrase should be in the client credentials, and will be in the /etc/backup.d/ jobs

Setup

Install necessary packages, currently the best way to do that is by using jessie-backports.

aptitude -t jessie-backports install python3-msgpack borgbackup

The files borg, borglocal, borg.helper, and borglocal.helper should be placed in /usr/share/backupninja, permissions 644, owned by root:root.

cd /usr/share/backupninja
wget https://redmine.palantetech.coop/attachments/download/7133/borg && wget https://redmine.palantetech.coop/attachments/download/7135/borg.helper && wget https://redmine.palantetech.coop/attachments/download/7136/borglocal && wget https://redmine.palantetech.coop/attachments/download/7134/borglocal.helper
chown root:root borg* && chmod 644 borg*

The files 70-76 are example backupninja jobs, which would go in /etc/backup.d, permissions 600, owned by root:root.

cd /etc/backup.d
wget https://redmine.palantetech.coop/attachments/download/7130/70-local.borglocal
wget https://redmine.palantetech.coop/attachments/download/7132/71-mfpl.borg
wget https://redmine.palantetech.coop/attachments/download/7138/75-borglocalbackupcheck.sh
wget https://redmine.palantetech.coop/attachments/download/7137/76-borgbackupcheck.sh
chown root:root 7* && chmod 600 7*

Job 70 is for backing up to a usb drive, which will be mounted at the beginning of every job and unmounted at the end.
Necessary variables to change: device uuid, filesystem type, included and excluded files, directory to mount the usb drive to, passphrase

Job 71 is for backing up to either a local folder or a folder accessible by ssh on a remote host.
Necessary variables to change: included and excluded files, directory to put the backups in, host, ssh user, passphrase

Jobs 75 and 76 are for checking the viability of the job 70 and 71 backups respectively, and require coping over the variables from those jobs.

Please change the passphrase variable in these jobs to something other than "PASSPHRASE", or leave it empty, which will turn off encryption.

troubleshooting

There is a version of the borg handler now included in newer versions of backupninja.
That handler requires two changes to job 71:

documentation:
https://0xacab.org/riseuplabs/backupninja/merge_requests/1
https://labs.riseup.net/code/projects/backupninja
https://borgbackup.readthedocs.io/en/stable/


Bylaws

Find copies of our bylaws here in both PDF and DOC format.


Certificate of Incorporation


Updated about 9 years ago by Jon Goldberg

Choosing a webhost

Introduction

Your website and/or CiviCRM database must be placed on a computer that's always connected to the Internet in order to be accessible to your visitors and/or staff. While theoretically you can take an unused computer in your office and put the site on it, this has multiple drawbacks. First, if your office's Internet connection goes out, the site becomes unavailable to people outside your office. Second, you take on the responsibility of maintaining the hardware - if the computer breaks, your site is offline until you can fix it.

For these reasons, unless your organization is in a position to support these issues, it's recommended to lease space from a "web hosting company", aka a "webhost". The webhost will maintain the hardware and Internet connection for your site. For small organizations, your choices fall into three categories: Shared hosting, Virtual Private Servers, and Platform-as-a-Service.

Shared hosting

Your webhost will put your site on a server with the site of dozens or hundreds of other sites. This makes it inexpensive, but it tends to be slow, and if one of the other sites gets a spike in usage, it can slow down your site.

Here are the shared webhosts our cients oten use: Shared Webhost options

Virtual Private Servers (VPS)

With a VPS, you're given a private server with a guaranteed level of resources. Because it's your own server, you can set it up as you need, and because your resources are guaranteed, you aren't affected by the activities of others. VPS price is determined by the amount of resources you get, and also whether you pay the webhost to maintain the server for you. You can also purchase VPS management services from Palante.

Some clients on a very low budget will lease an unmanaged VPS, but only maintain it during emergencies. While a bit risky, several of our clients have gone years with this approach successfully.

Here are some VPS companies Palante uses: VPS webhost options

Platform as a Service (PaaS)

With PaaS, you can hire a company to manage not just the server, but the application running on the server - for instance, WPEngine for Wordpress installations, or Pantheon for Drupal and Wordpress. These choices tend to be a bit more expensive than general hosting, but can be helpful if you expect to see spikes in demand, since these platforms will automatically allocate (and charge for) resources as they are needed.


Updated over 5 years ago by Morgan Robinson

CiviCRM

Disabling the requirement for CVV (credit card security code)
Payment Processor Test Cards
Exporting Raiser's Edge for CiviCRM
Exporting Salsa for CiviCRM
Exporting Filemaker for CiviCRM
Fundraising Reports
Overrides for dev/staging sites
Amazon SES for CiviCRM
Constant Contact sync for CiviCRM
Set up local bounce processing
Topics for "CiviCRM 102" training
CiviCRM launch checklist
CiviCRM multi-site checklist
Shared folders for migration
Migrating CiviCRM to a new site
PayPal


Updated over 8 years ago by Jon Goldberg

CiviCRM Kettle Transforms

This is the documentation for the CiviCRM Kettle Transforms available on Github here.


Updated almost 9 years ago by Jon Goldberg

CiviCRM multi-site checklist

The first two steps have more detailed docs below.

Editing civicrm.settings.php

Protip: Did you remember to comment out CIVICRM_DOMAIN_ID and CIVICRM_UF_BASEURL?

If you're using a "true" multi-site, where each site has its own civicrm.settings.php, the documentation on the CiviCRM wiki will suffice. Even if you're using Wordpress or Domain Access, there's good documentation there (I know, I wrote most of it).

The trick is to write code that can determine your domain based on the URL. Here is an example of how to do that using Domain Access or Wordpress with subdomain multi-site:

switch ($_SERVER['SERVER_NAME']) {

case 'www.xxx.org':
case 'xxx.org':
  define( 'CIVICRM_DOMAIN_ID', 1 );
  define( 'CIVICRM_DOMAIN_GROUP_ID', 2);
  define( 'CIVICRM_DOMAIN_ORG_ID', 105383);
  define( 'CIVICRM_UF_BASEURL'      , 'http://www.xxx.org/' );
  $civicrm_setting['URL Preferences']['userFrameworkResourceURL'] = 'http://www.xxx.org/sites/all/modules/civicrm';
  break;

case 'cdp.xxx.org' :
  define( 'CIVICRM_DOMAIN_ID', 2 );
  define( 'CIVICRM_DOMAIN_GROUP_ID', 19);
  define( 'CIVICRM_DOMAIN_ORG_ID', 106726);
  define( 'CIVICRM_UF_BASEURL'      , 'http://cdp.xxx.org/' );
  $civicrm_setting['URL Preferences']['userFrameworkResourceURL'] = 'http://cdp.xxx.org/sites/all/modules/civicrm';
  break;
// etc.
}

Wordpress with subfolder multi-site is a little trickier, here's what I've got. Note that this allows you to set a POST variable when running cron (via wget) to specify the correct site:

$multi_site_path = explode("/", $_SERVER[REQUEST_URI]);
if ($multi_site_path[6] == "cron.php") {
    $multi_site_choice = $_POST["site"];
} else {
    $multi_site_choice = $multi_site_path[1];
}

switch ($multi_site_choice) {

case 'wp-admin':
case '':
case false:
  define( 'CIVICRM_DOMAIN_ID', 1 );
  define( 'CIVICRM_DOMAIN_GROUP_ID', 84);
  define( 'CIVICRM_DOMAIN_ORG_ID', 1);
  define( 'CIVICRM_UF_BASEURL'      , 'http://www.yyy.org/' );
  break;

case 'hcnmd':
  define( 'CIVICRM_DOMAIN_ID', 2 );
  define( 'CIVICRM_DOMAIN_GROUP_ID', 74);
  define( 'CIVICRM_DOMAIN_ORG_ID', 67459);
  define( 'CIVICRM_UF_BASEURL'      , 'http://www.yyy.org/hcnmd' );
  break;
// etc.
}

Cron with multiple sites

You need to run cron separately for each site to run its scheduled jobs. Here's a good example of how to set it up.

0,15,30,45 * * * * /usr/bin/wget --config=/home/members/xxx/sites/xxx.org/users/xxx/xxx.org/include/civicrm-wgetrc http://www.xxx.org/wp-content/plugins/civicrm/civicrm/bin/cron.php
1,16,31,46 * * * * /usr/bin/wget --config=/home/members/xxx/sites/xxx.org/users/xxx/xxx.org/include/civicrm-wgetrc-site2 http://www.xxx.org/wp-content/plugins/civicrm/civicrm/bin/cron.php
2,17,32,47 * * * * /usr/bin/wget --config=/home/members/xxx/sites/xxx.org/users/xxx/xxx.org/include/civicrm-wgetrc-site3 http://www.xxx.org/wp-content/plugins/civicrm/civicrm/bin/cron.php
3,18,33,48 * * * * /usr/bin/wget --config=/home/members/xxx/sites/xxx.org/users/xxx/xxx.org/include/civicrm-wgetrc-site4 http://www.xxx.org/wp-content/plugins/civicrm/civicrm/bin/cron.php
4,19,34,49 * * * * /usr/bin/wget --config=/home/members/xxx/sites/xxx.org/users/xxx/xxx.org/include/civicrm-wgetrc-site5 http://www.xxx.org/wp-content/plugins/civicrm/civicrm/bin/cron.php
5,20,35,50 * * * * /usr/bin/wget --config=/home/members/xxx/sites/xxx.org/users/xxx/xxx.org/include/civicrm-wgetrc-site6 http://www.xxx.org/wp-content/plugins/civicrm/civicrm/bin/cron.php

Note that each civicrm-wgetrc file has the "site" set differently in the post-data. E.g.:

post-data=name=civicron&pass=<redacted>&key=<redacted>&site=site5
output_document = -
quiet=on
timeout=1


Updated almost 9 years ago by Jon Goldberg

CiviCRM user story examples

User stories are short narrative examples, 3-4 sentences long, of a typical example of how you intend to use a CRM database. Here are some examples.

CiviCRM for donor management

CiviCRM for policy advocacy


Common Problems

Updated over 6 years ago by Jessie Lee

Lost Mysql root password

Reset method 1

Reset method 2

fail2ban locked me out


Compass

Maintaining projects with compass-based themes requires the ability to edit SCSS files and to recompile them to CSS. Be aware that any changes to CSS files (compiled output) will be lost when the stylesheets are recompiled, so you should always make changes to the SCSS files.

Compass is a tool for managing Sass stylesheet projects. Sass is a language that compiles to CSS and also the name of the compiler. Compass makes working with Sass outside of a Ruby project much easier, however it is entirely possible to work with a Compass project using just the Sass compiler. But truly it's not worth the trouble! You likely won't encounter a situation where you have sass but not compass and no permission to install compass.

Installing compass

sudo apt-get install rubygems
sudo gem install compass --pre

This installs all dependencies including Sass.

Our projects tend to use additional compass plugins, which are installed as Ruby gems. Here are the set I commonly rely on:

sudo gem install compass-rgbapng survivalkit foundation

If one of these things is missing, you will be told in a very straightforward way when you try to compile a project that something is missing:

benjamin@dev1:/var/www/dev/nlg/sites/all/themes/prudence$ /var/lib/gems/1.8/bin/compass compile
LoadError on line 31 of /usr/lib/ruby/1.8/rubygems/custom_require.rb: no such file to load -- survivalkit
Run with --trace to see the full backtrace

The proper response to that is:

sudo gem install survivalkit

Compass projects

I'm referring to SCSS which is a syntax of Sass that closely resembles CSS. From Wikipedia

Sass consists of two syntaxes. The original syntax, called "the indented syntax" uses a syntax similar to Haml. It uses indentation to separate code blocks and newline characters to separate rules. The newer syntax, "SCSS" uses block formatting like that of CSS. It uses braces to denote code blocks and semicolons to separate lines within a block. The indented syntax and SCSS files are traditionally given the extensions .sass and .scss respectively.

None of our projects use the Sass syntax. Both variants are fully supported by Compass and Sass.

Compass-based themes are compass projects. You can identify them by the "config.rb" file in the theme directory. This config.rb specifies the location of SCSS source files and the destination of compiled output. In almost every case, "css" is the name of the output directory and "sass" is the name of the source directory.

These directories are relative to the theme directory itself. If there's any confusion, view the config.rb file to find the exact names of the directories.

# Location of the theme's resources.
css_dir         = "css" 
sass_dir        = "sass" 
extensions_dir  = "sass-extensions" 
images_dir      = "images" 
javascripts_dir = "js" 
fonts_dir       = "fonts" 

In the "sass" directory, you will see the source files that end with "scss." Some of these files are prefixed with "_" which identifies them as "partials." This is a Ruby concept that appears throughout the ruby ecosystem. For sass, this simply means:

If you have a SCSS or Sass file that you want to import but don’t want to compile to a CSS file, you can add an underscore to the beginning of the filename. This will tell Sass not to compile it to a normal CSS file. You can then import these files without using the underscore.

Recompiling compass

From the root of the compass project (the Drupal theme), issue this command:

/var/lib/gems/1.8/bin/compass compile

Editing SCSS

The SCSS syntax allows you to use the same CSS rules you would normally use and they will be passed through to the compiled output. The main trouble you will find is finding the place to edit.

If you cannot find the place to make your correction, it's possible that the CSS you want to change is actually being generated by a mixin or a partial, which means the specific code you want to edit is not going to be in the source file with a similar filename to the compiled output.

In this case, you will need to enable line comments and debugging in Compass. In the config.rb file look for this statement:

line_comments = false

And change it to "true."

Then recompile the output and you will see things like this:

/* line 62, ../../../../../../../../lib/gems/1.8/gems/compass-0.13.alpha.4/frameworks/compass/stylesheets/compass/typography/lists/_horizontal-list.scss */
#social-media-links .pane-content ul li {
  list-style-image: none;
  list-style-type: none;
  margin-left: 0;
  white-space: nowrap;
  float: left;
  display: inline;
  padding-left: 4px;
  padding-right: 4px;
}

This indicates that this CSS block is generated using code from the horizontal list mixin from Compass. That's not something you should necessarily edit, but you can read the documentation on that mixin to understand why it's being used.

/* line 86, ../sass/_custom.scss */
.section-member #content .burr-flipped-content-inner a {
  color: black;
}

This indicates that the code is being generated based on something in line 86 of _custom.scss, a partial file in this project, which you can edit.

It is not recommended to leave debugging enabled when committing your compiled output! It creates a lot of "diff noise" that doesn't belong in the git repository. When you've debugged your code, disable the line comments and re-compile before committing.


Conflict Resolution Sheet template

This Conflict Resolution Sheet is based upon work of AORTA and The Icarus Project. Our collective is going to start using this as a tool to help us deal with conflicts and have better communication. Each member fills out this sheet, and we store that information in our online documentation, enabling us to start conversations about difficult issues in ways that work better for us as individuals.

Some good ways to communicate with me about a conflict include:

(ex. email me individually beforehand, create a discussion ticket beforehand, call me individually, chat with me)

Some bad ways to communicate with me about a conflict include:

(ex. bring it up for the first time in an in person group meeting, chat with me, call me)

Some ways that I behave when I am having trouble with conflict include:

(ex. interrupting others, trying to close the conversation prematurely, avoiding talking about it, raising my voice, being silent)

When I am having trouble with conflict some positive things that I can do for myself include:

(ex. take time away from the issue, take a walk, eat or drink)

When other people notice me having trouble with conflict, some positive things that they can do to support me include:

(ex. break down the issue into smaller parts, use formal consensus process, propose putting off making a decision)

Ways in which I am good at supporting others in dealing with a conflict include:

(ex. phone calls, taking a walk, talking someone down)


Updated about 8 years ago by Jon Goldberg

Constant Contact sync for CiviCRM

This is documentation for this extension: https://github.com/cividesk/com.cividesk.sync.constantcontact. I'm not getting paid to fix the bugs mentioned here or ensure the accuracy of what I'm saying, but I took these notes and I'm posting them publicly in case anyone else benefits from them.

to this:

$last_sync    = '2000-01-01 00:00:00';


Covid visit policy


Custom Mapping Style

OpenLayers makes it possible to style the data layers of our maps. This could be helpful for visual accuracy of data represented as well as distinguishing between data on separate layers but rendered together on a single map.

To create a new style go to the Styles tab on the OpenLayers administration section. This lists all the existing styles available. Click to Add a new one or Edit an existing one. Give the style a distinguishing administrative name, machine name and description if adding a new one. All the options available have help text below each field which describe what each setting does. A few notes:

The best course when first creating styles is trial and error to get familiar with each option.


Data Munging (session notes from NPDev 2017)

See https://devsummit.aspirationtech.org/index.php?title=Data_Munging for canonical notes which Jon plans to edit for accuracy.


Updated 11 months ago by Jamila Khan

Debian 10 to 11

Pre update check

Stretch check

Check if any sources still point to stretch

cd /etc/apt
grep -nr stretch .

If so, bring those up to buster and run updates first
https://redmine.palantetech.coop/projects/commons/wiki/Debian_9_to_10

Metapackage check

Check to make sure kernel metapackage is installed, not just specific kernel

dpkg -l "linux-image*" | grep ^ii | grep -i meta should have results
If not, install metapackage
https://www.debian.org/releases/bullseye/amd64/release-notes/ch-upgrading.en.html#kernel-metapackage

Purged package check

List and purge removed packages with config files remaining
https://www.debian.org/releases/bullseye/amd64/release-notes/ch-upgrading.en.html#purge-removed-packages

aptitude search '~c'
aptitude purge '~c'

Hold check

These commands should have no results

aptitude search "~ahold" 
dpkg --get-selections | grep 'hold$'

Update sources list

Check which sources exist that point to buster

cd /etc/apt
grep -nr buster .

Edit the main list, and any others that come up

vim /etc/apt/sources.list

replace buster with bullseye

:%s/buster/bullseye/g

replace bullseye/updates with bullseye-security

:%s/bullseye\/updates/bullseye-security/g

https://www.debian.org/releases/bullseye/amd64/release-notes/ch-information.en.html#security-archive

Upgrade the system

Update the sources

apt-get update

If you get a NO_PUBKEY error, see https://redmine.palantetech.coop/projects/commons/wiki/Debian_10_to_11#Common-problems

Check for Necessary Disk Space

apt-get -o APT::Get::Trivial-Only=true dist-upgrade

Minimal Upgrade

 apt-get upgrade

Full Upgrade

apt-get dist-upgrade

Change configs during full upgrade?

package change configs
nrpe no
sudoers no
journald no
backupninja no
nginx yes but recheck after
redis yes but recheck after
sshd_config yes but recheck after
glibc yes
logrotate.d/apache2 yes

Upgrade MySQL databases (if MySQL/MariaDB installed)

mysql_upgrade

Check that things are up

Check what packages were removed

Common problems

NO_PUBKEY during update

W: GPG error: https://apt.postgresql.org/pub/repos/apt bullseye-pgdg InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7FCC7D46ACCC4CF8

Add the key for the specified repository:
0 meat:/etc/apt# sudo gpg -a --export 7FCC7D46ACCC4CF8 | sudo apt-key add -

See https://askubuntu.com/questions/13065/how-do-i-fix-the-gpg-error-no-pubkey for more information

Backupninja changes

After update, backupninja likely needs to be patched
https://redmine.palantetech.coop/projects/pt/wiki/Icinga2#Patch-the-backupninja-binary

Borg backup jobs to May First need to have port = 2201 added to the destination section.


Updated 2 months ago by Hannah Atkinson

Debian 11 to 12

Pre update check

Stretch check

Check if any sources still point to buster

cd /etc/apt
grep -nr buster .

If so, bring those up to bullseye and run updates first
https://redmine.palantetech.coop/projects/commons/wiki/Debian_10_to_11

Metapackage check

Check to make sure kernel metapackage is installed, not just specific kernel

dpkg -l "linux-image*" | grep ^ii | grep -i meta

should have results

If not, install metapackage
https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.en.html#kernel-metapackage

Purged package check

List and purge removed packages with config files remaining
https://www.debian.org/releases/bookworm/amd64/release-notes/ch-upgrading.en.html#purge-removed-packages

aptitude search '~c'
aptitude purge '~c'

Hold check

These commands should have no results

aptitude search "~ahold" 
dpkg --get-selections | grep 'hold$'

Update sources list

Check which sources exist that point to bullseye

cd /etc/apt
grep -nr bullseye .

Edit the main list, and any others that come up

vim /etc/apt/sources.list

replace bullseye with bookworm

:%s/bullseye/bookworm/g

Good to check but should be done if upgraded from Debian 10 to 11
replace bullseye/updates with bookworm-security

:%s/bookworm\/updates/bookworm-security/g

https://www.debian.org/releases/bookworm/amd64/release-notes/ch-information.en.html#security-archive

Upgrade the system

Update the sources

apt-get update

Check for Necessary Disk Space

apt-get -o APT::Get::Trivial-Only=true dist-upgrade

Minimal Upgrade

 apt-get upgrade
if it asks whether to change /etc/apt/trusted.gpg.d/debian-archive-jessie-security-automatic.gpg, say yes
other things it asks to change, say no, keep the existing file

Full Upgrade

apt-get dist-upgrade
Change configs during full upgrade?
package change configs
nrpe no
sudoers no
journald no
backupninja no
nginx yes but recheck after
redis yes but recheck after
sshd_config yes but recheck after
glibc yes
logrotate.d/apache2 yes

Upgrade MySQL databases (if MySQL/MariaDB installed)

mysql_upgrade

Validation Testing and Auditing Changes

if a web server, make sure websites are up
if an infrastructure server, test and make sure all parts of the infrastructure are working properly
if a PTC server check the recovery plan entry for that server to make sure everything has recovered

Check what packages were removed

    cat /var/log/apt/history.log | grep Remove

copy that into ongoing updates documentation for records
Make sure there was nothing important in there
If there were important packages in there

Common problems

NO_PUBKEY during update

When issues are encountered they should be listed here to ease future troubleshooting!

Backupninja changes

After update, backupninja likely needs to be patched
https://redmine.palantetech.coop/projects/pt/wiki/Icinga2#Patch-the-backupninja-binary

Borg backup jobs to May First need to have port = 2201 added to the destination section.


Updated over 5 years ago by Jack Aponte

Debian 8 to 9

Update sources list

vim /etc/apt/sources.list
:%s/jessie/stretch/g

Upgrade the system

Check for Necessary Disk Space

apt-get -o APT::Get::Trivial-Only=true dist-upgrade

Update the sources

apt-get update

Minimal Upgrade

 apt-get upgrade

Full Upgrade

apt-get dist-upgrade

Upgrade MySQL databases (if MySQL/MariaDB installed)

mysql_upgrade

Common problems

iBackup

PT ticket #35194-19

header.pl and constants.pm are in the ibackup directory but perl on debian does not look in the local directory for @INC.
to fix copy both header.pl and Constants.pm into /etc/perl/

backupninja

Until backupninja gets the fixes pushed out to the Debian repo (PT ticket #35353), errors like these may be reported by backupninja:

    CRITICAL - 38 errors or warnings
May 27 01:00:03 Warning: The partition table for /dev/ram0 could not be saved.
May 27 01:00:03 Warning: The partition table for /dev/ram1 could not be saved.
May 27 01:00:03 Warning: The partition table for /dev/ram2 could not be saved.

Resolution: add partitions = no to job 15/in 15-info.sys.

samba-ad-dc

if Samba is in AD-DC mode, it and winbind fail.
do these commands, then try to run the upgrade again

systemctl stop smbd nmbd winbind
systemctl disable smbd nmbd winbind
systemctl unmask samba-ad-dc
systemctl start samba-ad-dc
systemctl enable samba-ad-dc

https://lists.samba.org/archive/samba/2017-July/209972.html

Removing old versions of interpreters.

PHP5 (and Ruby 2.1) are not removed by default and will still be used if loaded!
If there is no need for these things to exist remove them with apt-get uninstall.

https://www.howtoforge.com/tutorial/how-to-upgrade-debian-8-jessie-to-9-stretch/


Updated almost 4 years ago by Jamila Khan

Debian 9 to 10

Update sources list

check which sources exist that point to stretch

cd /etc/apt
grep -nr stretch .

Edit the main list, and any others that come up

vim /etc/apt/sources.list
:%s/stretch/buster/g

Upgrade the system

Check for Necessary Disk Space

apt-get -o APT::Get::Trivial-Only=true dist-upgrade

Update the sources

apt-get update

Minimal Upgrade

 apt-get upgrade

Full Upgrade

apt-get dist-upgrade

Upgrade MySQL databases (if MySQL/MariaDB installed)

mysql_upgrade

Check that things are up

Check what packages were removed

Common problems

borg

mysql backup checking

Newest version of mariadb doesn't have separate databases for information schema or performance schema, so remove the old backups of those.

cd /var/backups/mysql/sqldump
rm information_schema.sql.gz
rm performance_schema.sql.gz

Icinga2

Need to re-replace the backupninja binary in /usr/sbin/backupninja https://redmine.palantetech.coop/projects/pt/wiki/Icinga2#Backupninja-monitoring

bang check

If using a script to check bang via wp, make sure the script uses the full path to /usr/local/bin/wp


DEPRECATED!

Cloning a new local site from live

First create databases on local environment.

mysqladmin -u root create PROJECT_drupal
mysqladmin -u root create PROJECT_civicrm

Then create your project Drupal root and install Drupal.

cd ~/workspace/PROJECT
drush --yes site-install -y --db-url=mysql://root@localhost/PROJECT_drupal

Create a Drupal alias of this project.

cd sites/default
drush sa --with-optional --with-db --full --alias-name=@local @self >> ~/.drush/PROJECT.aliases.drushrc.php

Log in to remote live host and create a new alias description for the live environment.

cd PROJECT DRUPAL ROOT
cd sites/default
drush sa --with-optional --with-db --full --alias-name=@live @self

Use these commands on localhost to synchronize.

drush --yes sql-sync @PROJECT.live @PROJECT.local
drush --yes rsync @PROJECT.live:%files @PROJECT.local:%files
drush @PROJECT.live civicrm-sql-dump | drush @PROJECT.local civicrm-sql-cli

Development environment

Goals

One Drupal installation can support many separate sites through Drupal's multi-site functionality. We want you to use this multi-site set up because it imposes some constraints on your development environment that will ensure that your work will migrate to other environments without a hitch.

Developing in a multi-site environment immediately shakes out a common class of problems related to paths and URLs. If the site you're developing is hosted in sites/mysite but in other environments it's hosted in sites/myhost.com, you'll have to use url() and drupal_get_path() functions to reliably load PHP include files or theme assets across all of your environments.

Drupal-specific deployment systems (AEgir) rely on sites' ability to deploy into a multi-site environment.

Finally, the development environment you build will be one that suits your skills and needs. There are minimums you need to accomplish, but we will indicate when you can stop if you're satisfied for now. If you want to build a more sophisticated, complex and automated development environment, keep following along.

Hacking core

Relying on a multisite-oriented environment enforces another good Drupal developer practice: don't hack core. If your Drupal projects share a common Drupal core, any core hacks you make will also be shared by all of these other projects. If a core hack is good enough for multiple sites, it's good enough to submit as a patch to core!

At the same time, it's justified to make some patches to core just for your development environment. For example, see the section about Xdebug.

Even though Devel module will override core's minified version of jQuery with an uncompressed copy, serious jQuery debugging will require that all of your scripts be uncompressed, which means you will need to replace jQuery UI with an uncompressed version.

Multisite : Each Drupal platform (core and any install profiles) will support

Uniformity : We explain how to set up a development environment which is uniform regardless of your preferred operating system. Only installation and upgrades are different depending on your operating system. Configuration and usage are consistent regardless of whether you use Windows, Mac OS X or Linux.

Maintenance : The development environment will be set up so that all of your web development projects are stored in the same part of your file system: a directory named workspace.

TODO: Create a directory inside your home directory named “workspace”

Credit

This development environment is directly inspired by the “sandbox” design used by CivicActions. This was developed by Owen Barton, Fen Labalme and the staff of OpenWall. The same principles for organizing the development environment are used for easily deploying sites in staging, testing and production environments by CivicActions.

Drubuntu also inherits from this design, and it's highly recommended for users who are already using Ubuntu 10.10.

You need to create a LAMP stack on your computer. LAMP is an acronym for Linux, Apache HTTP Server, MySQL and Perl/PHP/Python. Most Drupal sites are deployed on this software.

Drupal can run in other server environments and other platforms. Obviously, we are not going to insist that you switch your personal operating system to Linux just to develop Drupal sites, so Windows and Mac can be substituted for Linux. Solaris and FreeBSD are also suitable platforms for Drupal, but if you are developing sites in this environment, you will need to translate the Linux instructions using your own know-how. On Windows, IIS is supported, but it's not recommended. Drupal 7 supports MySQL, PostgreSQL and SQLite, but most Drupal sites use MySQL and there's a tremendous amount of documentation, GUI client software and support for MySQL. We're going to omit Perl and Python, because Drupal only needs PHP. You will still be able to install Perl or Python if you need it.

Platform specific

The main differences about configuring your development environment are the location of configuration files and the default settings in those configuration files. Use the table to identify the configuration files you may need to edit. If your files are located in another spot, note this because you'll need this information in the future.

MAMP Homebrew Windows Ubuntu Linux
http.conf /Applications/MAMP/conf/apache/httpd.conf /private/etc/apache2/httpd.conf /etc/apache2/httpd.conf
my.cnf /Applications/MAMP/db/mysql/my.cnf /usr/local/var/mysql /var/lib/db/mysql
php.ini /Applications/MAMP/conf/php5.3/php.inii /opt/local/etc/php5/php.ini /etc/php5/conf.d/php.ini

Some of us use version control to track changes and maintain a backup history for our configuration files

Windows

For Windows, we will use WampServer. http://www.wampserver.com/en/

WampServer 2.1 meets all the requirements for Drupal 7. For earlier versions of Drupal, you will need to also download an older version of PHP to install.

http://en.wikipedia.org/wiki/Comparison&#95;of&#95;WAMPs

WAMP's default httpd.conf needs to have mod rewrite enabled.

Wrap paths in quotation marks.

WAMP already loads configuration files from an aliases directory, so instead of including the file in httpd.conf, copy it to the directory named in the include.

Vhost_alias_module needed.

Mac

For Mac, there are two strategies for maintaining a Drupal development environment with their own strengths and weaknesses. Mac OS X could support Drupal on a the default system distributed by Apple Inc. It has Apache HTTPD Server and PHP already installed, and each user has a home directory named “Sites” which is Apache's user home directory. However, it does not have MySQL.

The Macintosh software community has provided two kinds of LAMP stacks: binary application packages and package managers. MAMP is an application package, which you download and install by copying. It is by far the quickest and easiest way to set up a development environment. However, it requires configuration after it has been installed. When you upgrade MAMP, you will need to remember to revisit these instructions so that your development environment does not break.

The alternative and more complicated method relies on a package manager. All package managers require that you install the Mac OS X Developer Tools. These developer tools are distributed on Mac OS X Install DVDs and also downloadable from http://developer.apple.com. We encourage you to customize the installation so that you can exclude documentation or extras you don't want, because the developer tools are very large. The documentation does include UNIX manual pages and Apple Developer documentation for Safari and Mobile Safari and a useful amount of web standards documentation. Likewise, XCode could be an IDE for your Drupal sites, but it's missing some simple features available in free and better tools.

As you improve in your web development skills in Drupal, you will confront the need to compile any number of new tools or servers for your environment. People in the Drupal community use more than just Drupal, and we will introduce you to new things like Compass, Solr, Memcached, Cucumber or a PECL add-ons. We urge you to use a package manager for these other tools because otherwise you risk destroying or disabling your development environment whenever you upgrade Mac OS X. Not every piece of useful web development software is distributed as a compiled binary for Mac OS X.

MAMP

MAMP from http://www.mamp.info/ is an application that you will copy into your Applications folder. Although it is running system services, it's attaching them to high-number, non-standard ports. MAMP's preferences are user specific, but MAMP may not run correctly if multiple users on the same computer want to use MAMP. It can run concurrently with Mac OS X's own Apache server (the “Web server” controlled by the Sharing preference pane), though you hopefully do not need to run both at the same time. MAMP.png

After copying MAMP to your Applications folder, run it. You will be greeted by the MAMP application window and the MAMP start page in your default web browser as served by MAMP. The start page for MAMP gives you access to phpMyAdmin, various PHP info pages and some documentation. It also shows the default MYSQL credentials.

MAMP runs as the logged in user, and any files it creates (MySQL database files, log files, files uploaded through your Drupal development sites, Imagecache generated files) will be owned by the logged in users. This makes backup very simple and reliable, but it could lead to problems if two users on the same computer want to use MAMP for their own projects.

The MAMP application window allows you to turn on and off the servers, open the start page, and set preferences. By default, MAMP will start and stop all of its services whenever the MAMP application is opened or closed. We like these preferences for MAMP. MAMP currently is distributed with PHP 5.2 and 5.3, and you may need to switch between these PHP versions if you're supporting Drupal 6 or other PHP applications. For Drupal 7, all of MAMP's default settings are appropriate.

Homebrew

Homebrew is a Mac OS X package manager that compiles from source and leverages software already installed as part of Mac OS X. Apache HTTPD Server and PHP 5.3 are already installed on Mac OS X Snow Leopard. Mac OS X Leopard uses PHP 5.2, which is also supported in Drupal 7. The only piece missing is MySQL.

You must install the Mac OS X Developer Tools before attempting to set up a development environment with Homebrew. If you are already suing MacPorts or Fink, you should avoid running both Homebrew and another package manager.

Install Homebrew from http://mxcl.github.com/homebrew/

In the terminal, run “brew install mysql.”

Ubuntu Linux

Users of Ubuntu Linux are fortunate to have access to a high-quality LAMP from within their package manager (usually Synaptic). Using the package manager, install these packages and their dependencies:

apt-get install apache2 apache2-mpm-prefork mysql-server curl
libapache2-mod-php5 php5-cli php5-common php5-curl php5-dev php5-mcrypt
php5-mysql php5-sqlite php5-xdebug php5-xsl php-apc php-pear

Apache

Drupal prefers Apache HTTPD Server. Core Drupal features like clean URLs are usually supported with no additional configuration.

Because Apache sometimes runs as a system user (usually www, www-data, apache or httpd), you may need to edit a few configuration files as the superuser or Administrator.

For your environment, Apache virtual host configurations will define at least one Drupal platform in which individual sites will be developed. A Drupal platform is the set of files that constitute Drupal core, any installation profiles, and any shared modules, themes or libraries located in sites/all.

In your workspace, create a new directory named conf.d.

In this directory, create these files. These configuration files define virtual hosts for each Drupal platform in your workspace.

https://github.com/bangpound/webdev-workspace/blob/master/workspace/conf.d/000-default.conf

This sets up some blanket options that apply to every project in your workspace, so that essential URL rewriting and access permissions are set correctly.

https://github.com/bangpound/webdev-workspace/blob/master/workspace/conf.d/drupal-7.conf

This tells Apache about Drupal 7. When you access your development environment's sites through your web browser, you will be navigating a URL that only your computer can access, for example http://new-site.7/. This configuration file tells Apache that any request it receives for a URL where the host name ends with .7, it should serve files from your Drupal 7 site directory.

https://github.com/bangpound/webdev-workspace/blob/master/workspace/conf.d/localhost.conf

This file gives you some flexibility to break out of the rigid structure of this development environment. If you need to engineer on new Drupal platform or maintain a site that for whatever reason does not function in a multisite environment, you can add that project to your workspace. For example, if you want to set up Tattler http://tattlerapp.com/, you can put its directory in your workspace as tattlerapp, and it will be accessed as http://tattlerapp.localhost/.

Typical, current Apache HTTPD installations use a modular configuration file arrangement. This allows us to modify httpd.conf as little as possible, because this may be rewritten in a software upgrade. In your Apache configuration directory /etc/apache2, there is httpd.conf and an extra directory.

In your Apache's httpd.conf, add the following:

bc. #
  1. Include workspace configurations #
    Include /PATH/TO/extra/webdev-workspace.conf

Then create the webdev-workspace.conf file in your extra directory. This simply loads the configuration files from your workspace.

#
# Include workspace configurations
#

Include /PATH/TO/YOUR/workspace/conf.d/*.conf

Hosts

The hosts file exists on every operating system http://en.wikipedia.org/wiki/Hosts&#95;(file), and it's usually the last place your computer looks to resolve a domain name. The hosts file is what gives your system the name “localhost,” and it's very important that you never remove entries from the file that you didn't put there yourself.

Each project in your development environment needs to be accessible at the root of a unique host name, because this is how your project will likely be deployed. If your project will be deployed in a subdirectory of a larger site or in a user's home directory, you will want to recreate this too so you can catch any path name errors.

Edit /etc/hosts

Add a block comment to indicate where your changes begin and add the first hostname.

bc. #
  1. Webdev-workspace project #
    127.0.0.1 d7

PHP

Usually when PHP is installed, it's set up for a development environment. Certain settings are unsuitable for a production environment, so you should make sure you have the right php.ini file installed.

Often, two configuration files come with PHP. php.ini-dist contains settings appropriate for your development environment. php.ini-recommended contains a few changes that hide error messages. The files have only a handful of differences to address the security and performance requirements of development and production environments.

Your development environment exists to show you how bad your work might be, and this is a better plan than letting your production environment show everyone in the world how bad your code actually is!

Verify that your php.ini file is configured with these settings:

bc. display_errors = On

PHP has a customizable error system, which Drupal 7 takes advantage of, which allows applications to handle non-fatal errors themselves and to change settings that affect output and thresholds.

Prior to Drupal 7, release versions of Drupal would not display PHP notice errors (E_NOTICE) and you either had to hack core or use a development release to display these error messages. Notices are useful because they reveal simple but hard to spot mistakes such as using variables that you've not assigned or using ambiguous syntax. http://drupal.org/node/34341

Error reporting may be annoying when the error has no apparent impact on the site's functionality, but notices often appear because of mistakes that are not going to cause fatal errors but which lead to buggy behavior.

MySQL

MySQL really has no special problems for setting up. Recommendations are to install from a package or build from source using a package manager.

Editing your server's my.conf file to enable innodb_file_per_table to make the InnoDB files smaller and fragmented, which is friendlier to a development environment.


Disabling the requirement for CVV

Per CRM-10366, the setting exists, but no form exists for it.

The easiest way to change it is via API explorer: <mysite>/civicrm/api/explorer

Select "Setting" as your entity, and "Create" as your action. Click the "CVV required for backoffice?" link that appears and enter a value of "0".


Drupal 8 and Backdrop Module Matrix

Updated over 7 years ago by Jack Aponte

This table is a list of Drupal 7 modules that Palante commonly uses in building client sites, accompanied by their current status in Drupal 8 and Backdrop.

Module Backdrop status Backdrop version D8 status D8 version
adaptivetheme unported unported pre-release dev
addanother ported full unported none
admin_views core n/a core n/a
auto_entitylabel pre-release full in progress dev
auto_nodetitle ported full in progress none
backup_migrate ported full in progress dev
better_formats ported full in progress dev
breakpoints unported n/a core n/a
calendar in progress dev in progress dev
civicrm in progress beta in progress beta
ckeditor_link unported n/a unported none
coffee ported full pre-release beta
comment_notify ported full in progress dev
ctools core n/a core & in progress* dev
date core n/a core n/a
devel ported full pre-release dev
diff unported none pre-release alpha
domain unported n/a in progress dev
entity core n/a core & in progress alpha
features unported none pre-release beta
google_analytics ported full ported full
honeypot ported full ported full
i18n in progress dev core n/a
imagefield_crop unported none unported none
imce ported full ported full
imce_filefield unported* none unported* none
imce_mkdir unported none unported n/a
imce_wysiwyg core n/a unported n/a
libraries ported full in progress dev
login_destination ported full unported n/a
logintoboggan ported full unported n/a
messaging unported none unported* none
metatag unported* none pre-release beta
nagios unported none pre-release dev
navbar core n/a core n/a
notifications unported n/a unported* none
og unported n/a in progress none
omega unported unported pre-release alpha
pathauto core full pre-release alpha
panels core (layouts) n/a pre-release beta
panels_breadcrumbs unported none unported none
picture unported none core n/a
piwik unported n/a ported full
radix_layouts in progress* dev pre-release rc
redirect ported full pre-release alpha
responsive_menus ported full in progress none
rules in progress dev pre-release alpha
search_api unported n/a pre-release alpha
search404 ported full pre-release alpha
service_links unported n/a unported none
sharethis ported full pre-release beta
spamspan unported* none pre-release dev
stage_file_proxy unported n/a in progress dev
strongarm core n/a core n/a
styleguide ported dev unported n/a
submitted_by ported dev unported n/a
token in progress dev pre-release alpha
total_control unported none unported none
views core n/a core n/a
views_bulk_operations* unported n/a in progress none
webform ported full in progress* none
webform_civicrm unported none planned none
wysiwyg core n/a core n/a
wysiwyg_mediaembed unported* none unported none

Notes

Also check out these related resources:


Drush


Updated about 8 years ago by Jon Goldberg

Exporting Filemaker for CiviCRM

Filemaker data is obviously a lot less uniform than data from another CRM, but I'm documenting some tips and tricks here.

File encoding

Specify in Kettle that input files exported as "Windows (ANSI)" (the default on Windows) are opened as file encoding CP-1252.

Converting vertical tabs

Here's a Javascript snippet I use in Kettle to convert them to separation characters:

var Activities;

Activities = replace(Activities,'\v', '\x01');

Note that sometimes I'll want to convert them to newlines instead.


Updated over 6 years ago by Jon Goldberg

Exporting Raiser's Edge for CiviCRM

There are two basic approaches to exporting RE data. There's the built-in export tool, and there's direct SQL interaction. This document will try to cover both approaches where possible. The Export tool has a lower barrier to entry, but a) there's some data you can't export with the tool, and b) the data will be denormalized, requiring additional transformation compared to extracting normalized SQL data.

Note that there's a good video on this topic by Young-Jin from Emphanos, LLC, here: http://sf2013.civicrm.org/migrating-raisers-edge-civicrm

Do you use Pentaho Kettle?

If so, you can use my Raiser's Edge to CiviCRM transforms, available here: https://github.com/PalanteJon/civicrm_kettle_transforms

Export tool - general guide.

The Raiser's Edge Export tool is on the left toolbar when you first enter Raiser's Edge.

From the tool, you will create a number of exports. When you first create an export, you'll be asked a number of questions, including Export Type (Constituent, Gift, etc.), a checkbox to include inactive records (check this), and an export file type (select CSV).

For most export, select Constituent as the Export type. This is the "base table" - all records will be joined relative to it.

Constituent Based Exports

Contact Information

RE differentiates between constituents and non-constituents in their system. If you create a new contact, they're a constituent - but then you might decide to add a spouse or employer record, which is NOT considered a constituent, and doesn't show up in most queries. Notably, non-constituents aren't exported when using the Export tool and your base table is "Constituent".

SQL

If extracting directly from SQL, SELECT * FROM RECORDS.

Note that you can extract only constituents by adding WHERE IS_CONSTITUENT = -1. For a Civi migration, I recommend importing all contacts.

Export tool (NOTE: This ONLY gets constituents).

Tab 1. General:
- Include all records.
- Head of Household processing: Export both constituents separately.
- Check all of the "Include these Constitutents": Inactive, deceased, no valid address

Tab 2: Output.
First, expand the "Constituent Information" in the left pane, and add every field to the export. Do the export (as a CSV).

Constituent Codes

In RE: Found at the bottom of the "Bio 2" tab.
In SQL: CONSTITUENT_CODES maps to "GroupContact". TABLEENTRIES stores the codes ("groups"). In my case, SELECT * FROM [CCR_July_snapshot].[dbo].[TABLEENTRIES] WHERE [CODETABLESID] = 43 did the trick. YMMV - see "deciphering stored procedures" below.

Export as one to many, below.
These map to "groups" in Civi - can also be mapped to "tags" if you don't need to track the begin/end date on them.

No need to export these fields:
System Record ID
Import ID
As of Civi 4.4.6, there's no way to import Group Begin/End dates via API, you need to do it via direct SQL.

Solicit Codes

These can map to groups - but also may map to privacy preferences or custom fields (e.g. Email Only, Do Not Solicit)

SQL to extract solicit codes:

SELECT RECORDSID AS external_identifier, LONGDESCRIPTION as solicit_code FROM CONSTITUENT_SOLICITCODES JOIN TABLEENTRIES ON SOLICIT_CODE = TABLEENTRIES.TABLEENTRIESID WHERE TABLEENTRIES.ACTIVE = -1

In my copy of RE, the CODETABLESID is 5044, so to get a list of all solicit codes, use:

SELECT LONGDESCRIPTION, ACTIVE FROM TABLEENTRIES WHERE CODETABLESID = 5044 ORDER BY SEQUENCE;

Addresses

SQL tables: ADDRESS, CONSTIT_ADDRESS

Addresses are a many-to-many relationship in RE.
Not all addresses in the database are visible in RE. Addresses where the INDICATOR field is 1 or 7, for instance. Make sure to look your data over and filter those out accordingly.

Phones/E-mail/websites

RE is a child of the 90's, so a) phones are tied to addresses, not contacts, and b) e-mails and websites are a type of phone.

Notes:

This SQL gets me a useful list of phones and e-mail for further processing in Kettle:

SELECT DISTINCT
  CONSTITADDRESSID
, CONSTIT_ID
, PHONETYPEID
, CONSTIT_ADDRESS_PHONES."SEQUENCE" 
, NUM
, DO_NOT_CALL
, TEXT_MSG
FROM CONSTIT_ADDRESS_PHONES
LEFT JOIN PHONES ON CONSTIT_ADDRESS_PHONES.PHONESID = PHONES.PHONESID
LEFT JOIN CONSTIT_ADDRESS ON CONSTITADDRESSID = CONSTIT_ADDRESS.ID

Relationships

Relevant SQL table: CONSTIT_RELATIONSHIPS

Relationships are different in Civi and RE in the following significant ways: Because relationships aren't necessarily reciprocal, I find it helpful to take my list of invalid relationships and do BOTH of the following:

Solicitor Relationships

Solicitor relationships are stored in a different table. I used this SQL to extract them:

SELECT
CONSTIT_ID
, SOLICITOR_ID
, TABLEENTRIES.LONGDESCRIPTION as solicitor_type
, AMOUNT
, NOTES
, cs."SEQUENCE" as weight
FROM CONSTIT_SOLICITORS cs
LEFT JOIN TABLEENTRIES ON cs.SOLICITOR_TYPE = TABLEENTRIES.TABLEENTRIESID
ORDER BY weight

Attributes

Attributes are the RE equivalent of custom fields. However, unlike custom fields, they can also have a "date" value and a "comments" value. While this can be replicated in Civi via multi-record custom field groups, ideally the data is evaluated attribute by attribute.

Valuable information about the setup of the attributes is available in RE from Config > Attributes.

Here's my preliminary SQL to export attributes from RE:

SELECT
ca.PARENTID as external_identifier
, ca.ATTRIBUTETYPESID
, at.DESCRIPTION as Category
, TABLEENTRIES.LONGDESCRIPTION as Description
, TEXT
, NUM
, DATETIME
, CURRENCY
, "BOOLEAN" 
, COMMENTS
, ca.ATTRIBUTEDATE
FROM ConstituentAttributes ca
JOIN AttributeTypes at ON ca.ATTRIBUTETYPESID = at.ATTRIBUTETYPESID
LEFT JOIN TABLEENTRIES ON ca.TABLEENTRIESID = TABLEENTRIES.TABLEENTRIESID

note: In the SQL above, "PARENTID" and not "ConstitID" is the correct foreign key to link this to the contact.

To get a list of option values out of RE for the attributes, use this SQL:

SELECT
DESCRIPTION
, at.CODETABLESID
, LONGDESCRIPTION
FROM TABLEENTRIES te 
LEFT JOIN AttributeTypes at ON te.CODETABLESID = at.CODETABLESID
ORDER BY DESCRIPTION

Attributes can be multi-record custom fields by their nature, so you have to account for that. Here's some alpha-grade SQL for sussing out which fields have multi-record custom fields:

SELECT ATTRIBUTETYPESID, PARENTID, COUNT(LONGDESCRIPTION)
FROM ConstituentAttributes ca
JOIN TABLEENTRIES te ON ca.TABLEENTRIESID = te.TABLEENTRIESID
GROUP BY PARENTID, ATTRIBUTETYPESID
HAVING COUNT(LONGDESCRIPTION) > 1
ORDER BY ATTRIBUTETYPESID

note: In Civi 4.5+, you could conceivable use "EntityRef" functionality to facilitate chained selects of OptionValue lists. That would let you create a multi-record custom field group that would very closely map how Attributes work in RE - but you'd have all the disadvantages of multi-record custom fields.

Salutations/addressee info

RE stores contact salutations and addressee info in two places.

Primary salutations/addressess are stored on the RECORDS table. PRIMARY_ADDRESSEE_ID, PRIMARY_ADDRESSEE, and PRIMARY_ADDRESSEE_EDIT, PRIMARY_SALUTATION_ID, PRIMARY_SALUTATION, PRIMARY_SALUTATION_EDIT.

An unlimited number of non-primary salutations can be stored in the CONSTITUENT_SALUTATION table.

Salutation options values are stored in the SALUTATION table, in the format "CODE1, CODE2, CODE3, etc.". Each code refers to an id in the SALUTATION_FIELDS table, which contains tokens (e.g. "First Name", "Spouse Last Name") as well as common words like "And".

Note that PRIMARY_ADDRESSEE is more akin to addressee_display in Civi, in that it stores the calculated display ID. Also note that when PRIMARY_ADDRESSEE_EDIT is -1 (true), that's the equivalent of a custom addressee in Civi, and the value stored in PRIMARY_ADDRESSEE_ID must be ignored.

Other constituent tables:

Skip these tables:

Contribution-related exports

Contributions/Gifts

Contributions (in RE parlance: Gifts) are complicated beasts!

Here are some relevant database tables and their equivalent in Civi:
GIFT civicrm_contribution
GiftSplit civicrm_line_item
CAMPAIGN Roughly maps to Campaign. Your mapping may vary and/or include custom fields.
APPEAL Also roughly maps to Campaign (or Source). Your mapping may vary and/or include custom fields.
FUND Roughly maps to Financial Type, but you might choose to import as a custom field instead.

See "Campaigns, Appeals, Packages" below for more.

Finally, note that RE has a concept of an "Adjustment". If a gift has already been posted to the accounting software, you can't simply change the amount. You create an adjustment, which has the updated data, and the two gift records are linked via the AdjustmentId. This is also how pledge installments are written off (see "Pledges" below).

Note that gift type is hardcoded into a function called "TranslateGiftType) - so you may want to include that function in your SQL, e.g.:

SELECT
gs.GiftId
, g.CONSTIT_ID
, gs.Amount
, g.DTE as gift_date
, FUND.DESCRIPTION as fund
, CAMPAIGN.DESCRIPTION as campaign
, APPEAL.DESCRIPTION as appeal
, g.PAYMENT_TYPE
, g.ACKNOWLEDGE_FLAG
, g.CHECK_NUMBER
, g.CHECK_DATE
, g.BATCH_NUMBER
, g.ANONYMOUS
, gst.LONGDESCRIPTION as giftsubtype
, g.TYPE
, DBO.TranslateGiftType(g.TYPE) as type2
FROM GiftSplit gs
LEFT JOIN FUND on gs.FundId = FUND.id
LEFT JOIN APPEAL on gs.AppealId = APPEAL.id
LEFT JOIN CAMPAIGN on gs.CampaignId = CAMPAIGN.id 
LEFT JOIN GIFT g on gs.GiftId = g.ID
LEFT JOIN TABLEENTRIES gst on g.GIFTSUBTYPE = gst.TABLEENTRIESID

(See here: http://www.re-decoded.com/2013/07/payment-type-or-payment-method-id-reference/#more-714)

Payment Type is also hard-coded, it seems:
1 Cash
2 Personal Check
3 Business Check
4 Credit Card
5 Standing Order
6 Direct Debit
7 Voucher
8 Other

Soft Credits

Stored in GIFTSOFTCREDIT. RE does NOT have the concept of a soft credit type - which is fine.

SELECT
, GiftId
, ConstitId
, Amount
, 'Soft Credit' as soft_credit_type
FROM GiftSoftCredit

Solicitor, Gift

(Important! Gift solicitors are different from Contact Solicitors)

I imported these as soft credits, but a different TYPE of soft credit. Here's the SQL I used to get the data out of RE:

SELECT
ParentId as gift_id
, SolicitorId as soft_creditee_external_identifier
, Amount
, 'Solicitor' as soft_credit_type
FROM GiftSolicitor

In Honor/Memorial Of (aka Tributes)

As of CiviCRM 4.5, In Honor/Memorial of is considered a form of soft credit. In RE, they're still separate, and are called Tributes. The structure is a little more complex - the table structure is Constituent <-> Tribute <-> Gift_Tribute <-> Gift. Civi is Contact <-> Soft Credit <-> Contribution.

Here is some preliminary SQL that pulls tribute data suitable for transformation and import to Civi as ContributionSoft entities. Note that CiviCRM doesn't have a concept of a "Description" but does have the concept of a PCP Note, so I'm importing the description there - in the future, I could see the argument for Civi exposing the PCP Note as a description.

<pre>
SELECT
</pre>
gt.GIFT_ID
, gt.TRIBUTE_TYPE
, t.DESCRIPTION
, t.RECORDS_ID as tributee_extenal_identifier
, te.LONGDESCRIPTION as tribute_type
FROM GIFT_TRIBUTE gt
JOIN TRIBUTE t ON gt.TRIBUTE_ID = t.ID
LEFT JOIN TABLEENTRIES te on gt.TRIBUTE_TYPE = te.TABLEENTRIESID

Pledges

Here are the relevant tables and their equivalents in Civi:
GIFT
Installment
InstallmentPayment

GIFT is equivalent to civicrm_contribution AND to civicrm_pledge. Pledges and contributions are stored in the same table - so a pledge paid in six installments will have SEVEN records in the GIFT field. Many organizations will specify a pledge in the Gift Type field - you can also tell by the presence of the INSTALLMENT_FREQUENCY, NUMBER_OF_INSTALLMENTS, FrequencyDescription, REMIND_FLAG, NextTransactionDate and the Schedule* fields. Note that some of these might also be used for recurring contributions.

Installment and InstallmentPayment are, when combined, the equivalent of the civicrm_pledge_payment table. civicrm_pledge_payment has a field scheduled_amount and actual_amount. RE's model is somewhat superior in that it allows partial payments on a pledge installment.

Notes: When creating pledges in CiviCRM via API, the open pledge payments are simultaneously created. To import pledge payments from RE, you first want to delete the auto-generated pledge payments, then import your own pledge payments. Finally, when importing pledge payments, the status of the parent pledge isn't updated. So you'll probably want some SQL to indicate whether the pledges are pending, complete or canceled, depending on the status of the pledge payments. Finally, watch out for adjustments, which is how some or all pledge installments might be marked as "Written Off" in RE.

The INSTALLMENT_FREQUENCY list is hard-coded:
1 Annually
2 Every 6 Months
3 Every 3 Months
4 Every 2 Months
5 Every Month
6 Due Twice/Month
9 Irregular
10 Single Installment

Some SQL:

/* Find all GIFT records with one or more associated Installment records.  These are pledges OR recurring gifts. */
SELECT DISTINCT
g.CONSTIT_ID
, g.ID as GiftId
, g.Amount
, g.DTE as receive_date
, FUND.DESCRIPTION as fund
, FUND.FUND_ID
, CAMPAIGN.DESCRIPTION as campaign
, APPEAL.DESCRIPTION as appeal
, g.PAYMENT_TYPE
, g.ACKNOWLEDGEDATE
, DBO.TranslateGiftType(g.TYPE) as type
, g.REF as note
,DATE_1ST_PAY
,g.DATEADDED
,g.DATECHANGED
,INSTALLMENT_FREQUENCY
,NUMBER_OF_INSTALLMENTS
,POST_DATE
,POST_STATUS
,REMIND_FLAG
,Schedule_Month
,Schedule_DayOfMonth
,Schedule_MonthlyDayOfWeek
,Schedule_Spacing
,Schedule_MonthlyType
,Schedule_MonthlyOrdinal
,Schedule_WeeklyDayOfWeek
,Schedule_DayOfMonth2
,Schedule_SMDayType1
,Schedule_SMDayType2
,NextTransactionDate
,Schedule_EndDate
,FrequencyDescription
, r.CONSTITUENT_ID
FROM Gift g
LEFT JOIN GiftSplit gs on g.ID = gs.GiftId
LEFT JOIN FUND on gs.FundId = FUND.id
LEFT JOIN APPEAL on gs.AppealId = APPEAL.id
LEFT JOIN CAMPAIGN on gs.CampaignId = CAMPAIGN.id 
LEFT JOIN RECORDS r ON g.CONSTIT_ID = r.ID
JOIN Installment i ON g.ID = i.PledgeId

Find pledge payments:

/* Find all pledge installments, and their related payments if they exist. */
SELECT
i.InstallmentId
, i.PledgeId
, i.AdjustmentId
, i.Amount as scheduled_amount
, i.Dte
, ip.Amount as actual_amount
, ip.PaymentId
, g.CONSTIT_ID
, g.RECEIPT_AMOUNT
, g.DTE as receive_date
, g.TYPE
, DBO.TranslateGiftType(g.TYPE) as type
FROM Installment i
LEFT JOIN InstallmentPayment ip ON i.InstallmentId = ip.InstallmentId
LEFT JOIN GIFT g ON ip.PaymentId = g.ID
/* Adjustments are stored in here too - when an adjustment happens, the pledge ID of the original value is blanked */
WHERE i.PledgeId IS NOT NULL 
ORDER BY i.AdjustmentId
/* Write-off Types: Covenant WriteOff, MG Write Off, Write Off */

Campaigns, Appeals, Packages

RE's model for campaigns is hierarchical and more sophisticated than CiviCRM's. A campaign (e.g. "Capital fundraising FY2017") can consist of several appeals (e.g. "Capital fundraising FY 2017 Spring Mailing"). Appeals will generally correspond to a particular action, especially a postal mailing. Campaigns and Appeals can be linked in a many-to-many relationship, but this is rare. The 1-to-many is by far the most common approach. Finally, an appeal can consist of "packages", which is a segment of your appeal. For instance, a single mailing ("appeal") could go out to major donors, regular donors and non-donors. You might also A/B test envelopes vs. postcards in the same mailing. This would result in a total of six "packages" for a single appeal. RE can track goals for each separately, and easily report on "number solicited" by package vs. number of gifts received.

Actions

Actions fill the same purpose as Activities in CiviCRM, but are architected quite differently - in some ways better, some ways worse. I don't have as much concrete info here, but here's a decent start at extracting Actions data via SQL:

SELECT
  a.ADDED_BY
, a.AUTO_REMIND
, a.RECORDS_ID as external_identifier
, cr.RELATION_ID as action_contact_id
, a.DTE as activity_date_time
, LETTER.LONGDESCRIPTION as letter
, a.PRIORITY as priority_id
, a.REMIND_VALUE
, a.CATEGORY
, a.Completed
, a.COMPLETED_DATE
, a.FUND_ID
, a.FOLLOWUPTO_ID
, a.TRACKACTION_ID
, a.PhoneNumber as phone_number
, a.Remind_Frequency
, a.WORDDOCNAME
, a.APPEAL_ID
, a.APPEAL_LETTER_CODE
, a.OUTLOOK_EMAIL_SUBJECT
, STATUS.LONGDESCRIPTION as status
, TYPE.LONGDESCRIPTION as type
, LOCATION.LONGDESCRIPTION as location
, ActionNotepad.ActualNotes
, CAMPAIGN.DESCRIPTION as campaign
FROM ACTIONS a
LEFT JOIN TABLEENTRIES as STATUS ON a.STATUS = STATUS.TABLEENTRIESID
LEFT JOIN TABLEENTRIES as TYPE ON a.[TYPE] = [TYPE].TABLEENTRIESID
LEFT JOIN TABLEENTRIES as LOCATION ON a.[Location] = LOCATION.TABLEENTRIESID 
LEFT JOIN TABLEENTRIES as LETTER on a.[LETTER_CODE] = LETTER.TABLEENTRIESID
LEFT JOIN ActionNotepad ON a.ID = ActionNotepad.ParentId
LEFT JOIN CAMPAIGN on a.CAMPAIGN_ID = CAMPAIGN.id
LEFT JOIN CONSTIT_RELATIONSHIPS cr on a.CONTACT_ID = cr.ID

"Category" and "Action type" both roughly map to "Activity Type". Same for "status" and "COMPLETED" and "COMPLETED_DATE" mapping to "activity_status". RE lets you designate a related Campaign, Fund and Proposal; out of the box, Civi only supports Campaign. The auto-reminder is more flexible than you can get with scheduled reminders in Civi without getting very complicated. "Solicitors" can't be mapped to a contact reference lookup, because more than one can be stored.

Note: The SQL above presumes only one note per action. If you have multiple notes per action, the action will be represented with multiple records, one per associated note. I'll try to provide SQL for extracting the notes separately at a later date.

Action Notes

Action Notes are stored in their own table. This maps to "Details" on a Civi activity, but you can log multiple notes per action in RE. Here's the SQL I used to extract them in preparation:

SELECT
  NotesID
, Title
, Description
, Author
, ActualNotes
, ParentId
, NotepadDate
, TABLEENTRIES.LONGDESCRIPTION as Type
  FROM ActionNotepad
  LEFT JOIN TABLEENTRIES ON ActionNotepad.NoteTypeId = TABLEENTRIES.TABLEENTRIESID
ORDER BY ParentId, ActionNotepad."SEQUENCE" 

Events

Events are stored fairly similarly to CiviCRM, except:

Here's some SQL to pull in the most relevant data:

SELECT
  se.CAPACITY
, se.END_DATE
, se.ID
, se.NAME
, se.START_DATE
, se.DATE_ADDED
, te.LONGDESCRIPTION as activity_type
, se.INACTIVE
, se.DISPLAYONCALENDAR
, CAMPAIGN.DESCRIPTION as campaign
, se.DESCRIPTION
FROM SPECIAL_EVENT se
LEFT JOIN CAMPAIGN on se.CAMPAIGN_ID = CAMPAIGN.id 
LEFT JOIN TABLEENTRIES te ON se.TYPEID = te.TABLEENTRIESID

Notes

RE notes (stored in the "ConstituentNotepad" table) can store quite a bit of data that Civi notes can not. They can store formatting (but with proprietary format tags, not HTML), inline photos, etc, and contain fields for date of note (separate from "Date Added" and "Date Changed"), the type of note, etc. Fortunately, they store plain-text versions of formatted notes in their own field. "Notes" is formatted; "ActualNotes" is plain text (except, well, where it isn't).

I've resolved this by removing notes over a certain length (above 15K and I assume you're a photo) and concatenating the fields I want to keep (e.g. Note Type and Description) with the ActualNotes field.

It may be possible to export the photos in the Notes using the techniques described below under "Media".

Here's the SQL I'm currently using to extract notes before doing transforms in Kettle:

SELECT
Title as subject
, Description
, Author
, ActualNotes
, ParentId
, cn.DateChanged
, LONGDESCRIPTION as NoteType
FROM ConstituentNotepad cn
LEFT JOIN TABLEENTRIES ON NoteTypeId = TABLEENTRIESID

Media

Exporting from RE

The files stored on the "Media" tab are held in the [dbo].[MEDIA] table in MS SQL. Assuming embedded and not linked data, the files are stored in the MS Access OLE format. It's relatively difficult to extract data from the OLE wrapper, though searching for extract access ole on any search engine will give you lots of options in a variety of languages. Blackbaud even has code to do it here, if you feel like using VBA.

I opted to use a commercial software package from Yohz Software called SQL Image Viewer. If you use this tool, enter the command:

SELECT OBJECT FROM [dbo].[MEDIA]

Then press "Execute Query", then press "Export" when it's done. This exports about 200 items/minute on a computer with a very slow hard drive.

If you want to decode the OLE containers on your own, there's the bcp CLI tool that installs with MS SQL, and you can run a SQL query from SQL Server Management Studio that extracts your OLE containers; see here. I experimented with this strategy, and this CLI command extracted an Access OLE container:

bcp "SELECT OBJECT FROM [July_Snapshot].[dbo].[media] WHERE ID = 1210 " queryout "C:\Users\Jon\Desktop\temp\test.ole" -T -N -S HOSTNAME\RE_EXPRESS

Importing into CiviCRM

The approach I took was to copy all the files into the "custom files" directory as specified in Administer > System Settings > Directories. Then I used the Attachment entity of the API to import the file to an activity. For the Media tab, I created activities especially to import the media onto.

Here's an example of the correct usage of the API to add in image "drill.jpg" to an activity with an ID of 628:

$result = civicrm_api3('Attachment', 'create', array(
  'sequential' => 1,
  'name' => "drill.jpg",
  'mime_type' => "image/jpeg",
  'entity_id' => 628,
  'entity_table' => "civicrm_activity",
  'options' => array('move-file' => "/home/jon/local/civicrm-buildkit/build/d46/sites/default/files/civicrm/custom/drill.jpg"),
));

Note that Civi will rename your files with random characters at the end, so this action is not idempotent. Keep a reserve copy of your exported RE media to roll back to!

If you use the API CSV import tool, your CSVs should look like this:

"name",entity_id,"entity_table","mime_type","options.move-file" 
"100.png",87511,"civicrm_activity","image/png","/home/jon/local/lccr/wp-content/plugins/files/civicrm/custom/100.png" 
"1000.pdf",88411,"civicrm_activity","application/pdf","/home/jon/local/lccr/wp-content/plugins/files/civicrm/custom/1000.pdf" 

And More

Tables that Civi doesn't have a direct counterpart for

Open each CSV file in Excel or similar. Sort each field by ascending AND descending to see if any data is stored in that field. If every record has no data or the same data, delete it - it's not being tracked in the current system. If you see only one or two records with a particular field, they're also probably fine to go, but check with the client first.

Next, strip out all of the constituent information except for primary/foreign keys. I like to keep in First/Middle/Last name just for human readability though. So leave in those three fields, plus any field with the word "ID" in it. This is your base constituent info, and will be in every other export you do.

Now comes the fun part! Export each table, one at a time, by adding those fields to an export that already includes the base constituent info.

For one-to-many relationships, the system will ask you how many instances of the information to export. I default to 12, then look over the data to see how many are actually used, then re-export with a higher or lower number.

I also remove records that don't contain the relevant data. For instance, when exporting Solicit Codes, I sort by the first Solicit Code. Then I scroll down past the folks that have Solicit Codes to those who have none, and delete the rows for folks who have none.

Note that for simplicity's sake, RE contains many views of the tables that, if you export them all, you'll have redundant data. There's no need to export "First Gift", "Last Gift", or "Largest Gift" - simply export all gifts. Likewise for "Preferred Address".

When exporting one-to-many tables that themselves contain one-to-many tables (e.g. Addresses contains Phones), do NOT select 12 of each! That means you're exporting 144 phone numbers per record. First determine the maximum number of addresses being tracked, re-export with that number, THEN export with phone numbers. Also, it's reasonable to export with 5 phone numbers per address.

NOTE: Letters sent is incomplete, there's more than 12 letters to some folks!

GIFTS is related to constituent on the last column (Constituent System Record ID)

Code Tables/Option Groups/Option Values

If you're extracting data from the SQL back-end, you'll see that the RE equivalent to Civi option groups is "code tables". There's two functions that handle lookups: dbo.GetTableEntryDescription and dbo.GetTableEntryDescSlim. To determine where the data is being accessed by the function, see "Deciphering MS SQL", below. Use the "lTableNumber" passed to those functions and you'll find your data in dbo.CODETABLES (comparable to civicrm_option_group), dbo.CODETABLEMAP and dbo.TABLEENTRIES (comparable to civicrm_option_value).

Deciphering MS SQL

SQL Server Profiler is a tool that lets you spy on SQL statements passed to MS SQL, which is good for determining where certain data lives. However, RE depends on functions and stored procedures, so sometimes the SQL won't tell you exactly where to look.

Looking Up Functions

These are embedded in SQL and have a nomenclature like: dbo.GetTableEntryDescSlim. Find them in SQL Server Management Studio: database > Programmability > Functions > Scalar-valued Functions.

Looking Up Stored Procedures

If, in the profiler, taking a certain action shows a command like this:
These have a syntax like:

exec sp_execute 48,43,'Acknowledgee'

You're dealing with a stored procedure. You need to find the corresponding exec sp_prepexec command (in this case, the one with a 48). In this case, it looks like:

declare @p1 int
set @p1=48
exec sp_prepexec @p1 output,N'@P1 int,@P2 varchar(255)',N'SELECT  Top 1 TABLEENTRIESID  FROM DBO.TABLEENTRIES WHERE CODETABLESID = @P1 AND LONGDESCRIPTION = @P2  ',43,'Acknowledgee'
select @p1

Note that there's a tool called "SQL Hunting Dog", a free plug-in for SQL Server Management Studio, which makes locating stored procedures, etc. easier.

Addressee/Postal Greeting/E-mail greeting

RE has a much wider variety of greeting formats out-of-the-box. The "spouse ID" is stored on the record to enable quick lookups of addressee greetings that include the spouse.

It's likely that you'll want to map existing RE greetings to Civi greetings. Here is some SQL that will show you how the current greetings in RE are constructed:

/****** Script for SelectTopNRows command from SSMS  ******/
SELECT s.ID
, sf1.FIELDNAME as FIELD1
, sf2.FIELDNAME as FIELD2
, sf3.FIELDNAME as FIELD3
, sf4.FIELDNAME as FIELD4
, sf5.FIELDNAME as FIELD5
, sf6.FIELDNAME as FIELD6
, sf7.FIELDNAME as FIELD7
, sf8.FIELDNAME as FIELD8
, sf9.FIELDNAME as FIELD9
, sf10.FIELDNAME as FIELD10     
, sf11.FIELDNAME as FIELD11
, sf12.FIELDNAME as FIELD12
, sf13.FIELDNAME as FIELD13
, sf14.FIELDNAME as FIELD14
, sf15.FIELDNAME as FIELD15
, sf16.FIELDNAME as FIELD16
, sf17.FIELDNAME as FIELD17
, sf18.FIELDNAME as FIELD18
, sf19.FIELDNAME as FIELD19
, sf20.FIELDNAME as FIELD20
  FROM SALUTATIONS s
  LEFT JOIN SALUTATION_FIELDS sf1 on CODE1 = sf1.ID
  LEFT JOIN SALUTATION_FIELDS sf2 on CODE2 = sf2.ID
  LEFT JOIN SALUTATION_FIELDS sf3 on CODE3 = sf3.ID
  LEFT JOIN SALUTATION_FIELDS sf4 on CODE4 = sf4.ID
  LEFT JOIN SALUTATION_FIELDS sf5 on CODE5 = sf5.ID
  LEFT JOIN SALUTATION_FIELDS sf6 on CODE6 = sf6.ID
  LEFT JOIN SALUTATION_FIELDS sf7 on CODE7 = sf7.ID
  LEFT JOIN SALUTATION_FIELDS sf8 on CODE8 = sf8.ID
  LEFT JOIN SALUTATION_FIELDS sf9 on CODE9 = sf9.ID
  LEFT JOIN SALUTATION_FIELDS sf10 on CODE10 = sf10.ID
  LEFT JOIN SALUTATION_FIELDS sf11 on CODE11 = sf11.ID
  LEFT JOIN SALUTATION_FIELDS sf12 on CODE12 = sf12.ID
  LEFT JOIN SALUTATION_FIELDS sf13 on CODE13 = sf13.ID
  LEFT JOIN SALUTATION_FIELDS sf14 on CODE14 = sf14.ID
  LEFT JOIN SALUTATION_FIELDS sf15 on CODE15 = sf15.ID
  LEFT JOIN SALUTATION_FIELDS sf16 on CODE16 = sf16.ID
  LEFT JOIN SALUTATION_FIELDS sf17 on CODE17 = sf17.ID
  LEFT JOIN SALUTATION_FIELDS sf18 on CODE18 = sf18.ID
  LEFT JOIN SALUTATION_FIELDS sf19 on CODE19 = sf19.ID
  LEFT JOIN SALUTATION_FIELDS sf20 on CODE20 = sf20.ID

Things I see that RE does better than Civi:


Updated about 8 years ago by Jon Goldberg

Exporting Salsa for CiviCRM

Do you use Pentaho Kettle?

If so, you can use my Salsa to CiviCRM transforms, available here: https://github.com/PalanteJon/civicrm_kettle_transforms

Before you begin

Exporting Supporter Data

When logged in as a Manager, go to "Supporter Management" tab, and "Query/Export" will give you a full export of contact data. There's a "Select My Entire List" button. There's also an "include my deleted supporters" checkbox, which you may want to check (but probably not). This will NOT get you groups/tags, but WILL get you the equivalent of civicrm_contact plus all custom fields attached to the contact.

Exporting Data not on the Contact Entity

You need to get to custom reports. If you have the Reports tab, great; if not, you can get there by clicking "Supporter Management" tab, then "Built-in Reports", then "Clone and Edit" a report, then select "List Your Custom Reports". Or go straight here: https://hq-org.salsalabs.com/dia/hq/reports/list.jsp?table=report

Groups

Tags

Events

As above, but no conditions, and report on the "Event" table. If you're only importing legacy events, this is a good list of easy fields to import:
Event KEY
Reference Name
Event Name
Description
Start
End
Deadline
Maximum Attendees
Date Created (is a useful proxy when "Start" isn't defined)

Participants

As above. Report on tables "Supporter Event", then "Event". Note that there are a lot of tables that look like "Supporter Event", like "Supporterevent" and "Supporter Events"!

Note that if you're matching on a unique field from "Events", you can just export "Supporter Event" and not the "Event" table.
Here are the fields I exported and what I matched them to:
Supporter KEY external_identifier
Event KEY event_key
Status status
Type role
Date Created register_date
Last Modified register_date

(I didn't actually map both the date fields to register_date. I use "Date Created" unless it's NULL.

Recurring Donations

Export on table "Recurring Donation". Here's the field mapping:

Recurring Donation Key -> trxn_id
Supporter Key -> external_identifier
Transaction date -> create_date
Start Date -> start_date
RPREF -> invoice_id
RESULT (exclude 37, they're tests)
Amount -> amount
Pay Period -> use to remap to frequency_unit and frequency_interval
TERM -> installments
Status(?)

Note that in Salsa, there's no clear-cut way to tell whether a contact's recurring donation is still working or not - "Status" isn't a complete indicator, and thus doesn't cleanly map into Civi.

Donations

It's important to link the recurring donations to the donations when exporting if you need to identify the first recurring donation (e.g. to generate thank-you letters if it's a first donation and recurring).

Under "Conditions", filter by "Supporter KEY is not empty" - when Salsa imports legacy data into their system, sometimes they don't do such a hot job.

Export on table "Donation" linked to table "Donate Page". You'll want these fields (more or less):

Donation.donation KEY
Donation.supporter KEY
donate_page.Reference_Name (contribution_page_id)
Donation.Date_Entered
Donation.Transaction Date(donation)
Donation.amount(donation)
Donation.Transaction Type(donation) (financial_type)
Donation.RESULT (only 0 and -1 should be accepted, others are tests)
Donation.Tracking Code(donation) (source)
Donation.Designation Code(donation) (source - not sure what the differentiation between designation and tracking codes is)
Donation.In Honor Name(donation)
Donation.In Honor Email(donation)
Donation.In Honor Address(donation)
Recurring donation.Transaction Date(recurring_donation)
Donation.Order_Info
Donation.Form of Payment (payment_instrument)

Special Note: Salsa seems to keep donations that are tied to supporters that don't exist (that is, the supporter_key on the donation doesn't match anyone in Salsa). I've seen this multiple times; I don't know why this is, but this is normal.

Actions

Salsa has a separate table for actions, making them structurally comparable to Civi Events. A good match in Civi is the Survey entity, which is used for petitions.

Export as above. Report on table "Action".
Condition: "Reference Name" is not empty.

Fields:
Action.action KEY
Action.Reference Name
Action.Date Created
Action.Signatures
Action.Last Modified
Action.Title
Action.Description

Note: "Description" is likely to contain HTML that breaks the export. I find it helpful to export it as the last column.

Supporter_Actions

Export "action" and "supporter_action" and "supporter_action_comment" in that order.
Condition: supporter_key IS NOT NULL

Here's the fields I grabbed with their mapping:
supporter_action.supporter_key
action.Reference Name
supporter_action.Date_Created
supporter_action_comment.Comment
supporter_action.Last_Modified

(I didn't actually map both the date fields to activity_date_time. I use "Date Created" unless it's NULL.

Letter to the Editor

This isn't a feature CiviCRM has, but it maps reasonably well to an activity when migrating.

Date_Created -> activity_date_time
supporter_KEY -> external_identifier
person_media_ID? -> seems important - but I can't find a matching table, and this data doesn't seem to be accessible from Salsa.
Letter_Subject -> details (concatenated)
Letter_Content -> details
Reference_Name -> subject

Chapters

Programmatically creating custom fields in CiviCRM

To do this, you'll need to create a custom group, custom fields, option groups, and option values.

Custom Field Group

Don't export anything from Salsa - just create this manually, or using my Kettle transform.

Custom Fields, Option Groups, Option Values

Custom Columns

custom_column_key -> needed to relate to custom_column_options. Store in filter temporarily.
data_table (should = supporter_custom. In theory, we can probably export non-contact custom fields in this same process)
Date_Created -> created_date
label -> label
type -> data_type, html_type (can be bool, enum, text, varchar)
Description -> help_pre
Order -> weight
Name -> value*
custom_column_option_key -> value*
is_a_zero_index_enum -> (if this is true, use the "Name" as value. Else use custom_column_option_key as value).

Option Groups
These will be created automatically when you create custom fields.

Custom Column Options

custom_column_KEY -> a lookup field (against civicrm_custom_field.filter to get option_group_id)
custom_column_option_KEY -> value
value -> name
label -> label
isDefault -> is_default
isDisplayed -> is_active
_Order -> weight

Receive_Email field

Here's a TSV of how to interpret the "Receive_Email" field.
code meaning
-24 INACTIVE: (5.1.1 User Unknown)
-26 INACTIVE: Address contains RFC spec. invalid characters / is improperly formatted
-3 INACTIVE: UNSUBSCRIBED (actively unsubscribed by user)
-30 INACTIVE (Reported as Spam): Other Blacklist (BLACKLIST)
-32 (Unknown Status)
-35 INACTIVE (Reported as Spam): Outblaze
-4 (Unknown Status)
-42 INACTIVE (Recipient Initiated Spam Report): MSN/Hotmail/WebTV
-44 INACTIVE (Recipient Initiated Spam Report): UNTI (Juno/NetZero/FreeServers)
-45 INACTIVE (Recipient Initiated Spam Report): Yahoo
-46 INACTIVE (Recipient Initiated Spam Report): Comcast.net
-47 INACTIVE (Recipient Initiated Spam Report): AOL
-48 INACTIVE (Recipient Initiated Spam Report): RoadRunner
-50 ISP Specific Bounces: AIM.com Unactivated account
-51 INACTIVE (Recipient Initiated Spam Report): Excite
-52 INACTIVE (Reported as Spam): Earthlink
-53 (Unknown Status)
-54 (Unknown Status)
-60 Bad Address/Expired address, etc (n.b.: These addresses are all @democracyinaction.org)
-9 INACTIVE: DEATH (supporter has died)
0 0 - Inactive or Unknown(not subscribed)
1 1 - Imported or unknown (subscribed)
10 ACTIVE: CONFIRMED (Double opt-in)
3 ACTIVE: CLIENT (Client has directly signed up)

Fundraising Reports

  1. How many people were asked (invited)
  2. How many people made a gift (& what percent is that of those who were asked)
  3. How much money was raised (gross & net)
  4. What was the range of gifts (lowest – highest)
  5. How many people were involved in asking (& who – board members, other members, donors)
  6. How many gifts were from first-time donors
  7. What was the average gift/median gift
  8. How many people made a donation in whatever time period you want to look at
  9. How much money was raised (gross & net)
  10. How many gifts were made (different from # of people because some people make multiple donations)
  11. What is your retention rate, ie, of all the donors who gave in 2008, which of those people gave in 2009, expressed as a percentage (Norm is 65%)
  12. How many people gave for the first time in whatever time period you designate
  13. How many people gave “x” amount or more in a given time period
  14. What percent of the board gives $, and what percent is involved in fundraising
  15. What is your conversion rate: of all the people who gave for the FIRST time in 2008, which of those people gave in 2009, expressed as a percentage (norm is 25-35%).
  16. What percentage of donors increased the size of their gift/s in one year, and by what method were they asked?
  17. How many and what percentage of people give only on-line or only by mail
  18. How many people have given for 5 years (or any # of years you want to know about) or more, and who are they. This is really important information when you are ready to do something with it.
  19. How many donors are also actively involved in the organization (beyond giving money)
  20. Who knows who, that is, what are all the connections & relationships donors have to the organization

Updated about 9 years ago by Jamila Khan

General support changelog


Grievance Policy

1. What is a grievance? How are they delineated?

A grievance is a complaint about a specific injury, injustice, dissatisfaction or wrong decision made by a worker or workers. A grievance will fit into the following categories.

A. A claim that a decision violates existing Co-op policy.
B. A claim that a decision creates an unfair situation for which no policy exists.
C. A claim that a decision duly following Co-op policy creates an unfair situation because the policy itself is inherently unfair or discriminatory.

To determine the type of grievance, review the [[pt:Current Policies]] and [[pt:Chronological Decisions document]] wikis, and see if there is any reference to a relevant policy. If there is a policy and it has been broken, use grievance category A; if there is no policy, use grievance category B; if there is a policy and the policy is the problem, use grievance category C.

2. Form

The grievance form is in the PT Dropbox under Administrative\Grievance Policy and also available as a download here: PT grievance form.doc

3. Grievance decision making process

The decision making process varies depending on the category of grievance, but the base framework is the same.

A. For grievances that violate existing Co-op policy, the actions in question will be reviewed and the policy in question will remain static for the duration of the grievance process.

B & C. For grievances where no policy exists or where policy creates an unfair situation, all involved parties will work to craft a new policy, which will then follow the [[pt:Consensus Policy]] until there is consensus or an alternate resolution of all current worker-owners.

4. List of outside helpers

5. Who can make a grievance?

Anyone who at the time of the action in question was one of the following:

6. When can you make a grievance?


How to Enable Private Browsing

Private browsing is a way of using your web browser such that it doesn't leave a trace on the computer of any of the sites you visited. While this isn't an effective way to be anonymous on the Internet (see Private Browsing Myths), it's often helpful if you want to see what your website looks like to anonymous visitors.

To enable private browsing, please find your web browser below and click the link - or use the keyboard shortcut provided.
Firefox (Ctrl-Shift-P on Windows, Command-Option-P on Mac)
Chrome (Ctrl-Shift-N on Windows, Command-Shift-N on Mac)
Safari (Command-Shift-N on Mac)
Internet Explorer (Ctrl-Shift-P)

Sigh - I couldn't find this page beforehand when I looked, but I just found this: http://www.wikihow.com/Activate-Incognito-Mode


Last updated by Janine Ko Updated almost 4 years ago

How to Set Up Microsoft 365

Outlook Web Access (OWA)

Open someone else's mailbox

Add someone else's ecalendar

Mobile Devices

Android devices

Note: different phone makers may have slightly different procedures

iOS Devices

Desktop

Outlook 2007

Outlook 2016 and 2019

Refer to these instructions for Outlook for PC to add your own email account.

Outlook for Mac

Refer to these instructions for Outlook for Mac to add your own email account.
Refer to these instructions to open a shared or delegated inbox on Outlook desktop application for Mac. Make sure to check in with Palante about what level of access you have.


IBackup for linux

installing and configuring IBackup perl scripts

cd /usr/local/sbin
wget https://www.ibackup.com/online-backup-linux/downloads/download-for-linux/IBackup_for_Linux.zip
unzip IBackup_for_Linux.zip
cd IBackup_for_Linux/scripts
chmod a+x *.pl
./Account_Setting.pl
Enter your

It then asks if you want to log in, say yes.

setting up backupninja jobs

cd /etc/backup.d

create 56-iBackup.sh with contents:

#!/bin/sh

cd /usr/local/sbin/IBackup_for_Linux/

OUTPUT=$((

/usr/local/sbin/IBackup_for_Linux/Backup_Script.pl --silent

) 2>&1)

if [ $? -ne 0 ]
then
warning $OUTPUT
fi

create 57-iBackup_test.sh
remember to change the path to include your username

#!/bin/sh

## YOU MUST CHANGE THIS TO REFLECT THE LOCAL PATH
path="/usr/local/sbin/IBackup_for_Linux/user_profile/USERNAME/Backup/Manual/LOGS" 

# declare variables for checking the timestamp against
today=`date "+%a %b %e"`
yesterday=`date -d "yesterday" "+%a %b %e"`
cnt=`ls -l $path |grep "$today\|$yesterday"| wc -l`

# find if there are errors
errorcnt=`find $path -name **"$yesterday"** -exec cat {} \; | grep "failed to backup" |  rev | cut -d ' ' -f 1| rev`
errors=`find $path -name **"$yesterday"** -exec cat {} \; | grep -i -A 6 "Error Report"`

# for testing
#echo -e "$today \n $yesterday \n $cnt \n $errorcnt \n $errors" 

OUTPUT=$((

if [ $cnt -eq 0 ] ; then 
echo "CRITICAL - A backup did not run on $yesterday!" 
exit 2
fi

if [ $errorcnt -ne 0 ] ; then 
echo "CRITICAL - $errorcnt Errors in backup!\n$errors" 
exit 2

else echo "OK - Backup ran $yesterday with no errors" 
exit 0
fi

) 2>&1)

if [ $? -ne 0 ]
then
warning $OUTPUT
fi

For newer versions of the program:

#!/bin/sh

## YOU MUST CHANGE THIS TO REFLECT THE LOCAL PATH
#path="/usr/local/sbin/IBackup_for_Linux/user_profile/palantetech/Backup/Manual/LOGS" 
path="/usr/local/sbin/ibackup/user_profile/palantetech/Backup/Manual/LOGS" 

# declare variables for checking the timestamp against
#today=`date "+%a %b %e"`
#yesterday=`date -d "yesterday" "+%a %b %e"`
today=`date "+%s" | cut -c -4`
yesterday=`date -d "yesterday" "+%s" | cut -c -4`

cnt=`ls -l $path |grep "$today\|$yesterday"| wc -l`

# find if there are errors
errorcnt=`find $path -name **"$yesterday"** -exec cat {} \; | grep "failed to backup" |  rev | cut -d ' ' -f 1| rev`
errors=`find $path -name **"$yesterday"** -exec cat {} \; | grep -i -A 6 "Error Report"`

# for testing
#echo -e "$today \n $yesterday \n $cnt \n $errorcnt \n $errors" 

OUTPUT=$((

if [ $cnt -eq 0 ] ; then 
echo "CRITICAL - A backup did not run on $yesterday!" 
exit 2
fi

if [ $errorcnt -ne 0 ] ; then 
echo "CRITICAL - $errorcnt Errors in backup!\n$errors" 
exit 2

else echo "OK - Backup ran $yesterday with no errors" 
exit 0
fi

) 2>&1)

if [ $? -ne 0 ]
then
warning $OUTPUT
fi

Developed from
https://www.ibackup.com/online-backup-linux/readme.htm
https://hq.palantetech.coop/issues/25142


Icinga Skillshare

The monitoring software we use is Icinga, which is a fork of Nagios.

Our instance is at http://icinga.palantetech.com/icinga/, hosted on Ramen, the creds are in the credentials folder.

We currently have over 200 hosts, so the easiest place to look for what is happening is the tactical overview

Let's use Octavia as an example.
Here is the url for all the things being checked on octavia.
http://icinga.palantetech.com/cgi-bin/icinga/status.cgi?host=octavia

Now let's see how those are set up.
On Ramen, take a look at /etc/icinga/objects/ptc/octavia_ptc.cfg

This is the old way I was setting up checks.
I create a cfg file for each host, define the host, then define all the services for it.

Each service corresponds to a specific command definition (those live in /etc/nagios-plugins/config), which says a specific way to run a specific check script (those live in /usr/lib/nagios/plugins).

For example, the command definition for check_http is

# 'check_http' command definition
define command{
        command_name    check_http
        command_line    /usr/lib/nagios/plugins/check_http -H '$HOSTADDRESS$' -I '$HOSTADDRESS$'
        }

Which runs the script at /usr/lib/nagios/plugins/check_http with the -H and -I options both pointing to the $HOSTADDRESS$ variable pulled from the host definition (which for Octavia is octavia.mayfirst.org) then gives that back to Icinga which displays the results like so.

Now let's look at another host, nlg.org

If we look on Ramen at /etc/icinga/objects/clients/nlg/nlg.org_nlg.cfg, there are no checks defined!

That's because I've switched them over to the newer much much more efficient way.

The hosts can be put into nested hostgroups, and have services assigned to each group.

For example, the host nlg.org is in the drushbackups033 hostgroup, defined in /etc/icinga/objects/hostgroups/drushbackups.cfg.
That hostgroup is itself part of the drushbackups hostgroup, also defined in /etc/icinga/objects/hostgroups/drushbackups.cfg, the dnschecks hostgroup defined in /etc/icinga/objects/hostgroups/dns.cfg, and the domains hostgroup /etc/icinga/objects/hostgroups/domains.cfg.

Every host that is part of the domains hostgroup gets a service for expiration and blacklists, defined in /etc/icinga/objects/services/domain.cfg; if it's in dns, it gets that service, and if it's in a drushbackup### group, it gets drush backup checks for Drupal, Civi, and Default backups in number corresponding to the first, second, and third of the numbers at the end of the group (/etc/icinga/objects/services/drushbackup.cfg), as well as checking the oldest backup (still in progress).

Q&A


Updated over 1 year ago by Jack Aponte

Installing and Updating Drush

An important note: we no longer need to install any version of Drush except Drush 8, since that now supports Drupal 6, 7, <8.3 and Backdrop.

Installing and updating Drush 8 for all users

These instructions follow the "Install a global Drush via Composer" instructions under in the Drush 8 alternative installation documentation.

If needed, follow this Composer documentation to install or update Composer on the server.

Since running composer install as root is a security risk, you can use commands like these to install or update Drush in your home directory first, then move it to a globally-accessible location using sudo:

cd ~
COMPOSER_HOME=$HOME/drush COMPOSER_VENDOR_DIR=$HOME/drush/8 composer require drush/drush:8.4.11
cd ~/drush/8/drush/drush
composer install
cd ~
sudo rm -rf /opt/drush; sudo mv ~/drush /opt/
sudo chown -R root: /opt/drush
sudo rm /usr/local/bin/drush; sudo ln -s /opt/drush/8/drush/drush/drush /usr/local/bin/drush

Replace "8.4.11" (last updated in this documentation on 12/04/2022) with the tag of the most recent stable 8.x release (see https://github.com/drush-ops/drush/releases)

If you must run composer install and the other commands above as root, e.g. if that's the only user we have access to on the server, you can do so by changing the COMPOSER_HOME path to /opt/drush and COMPOSER_VENDOR_DIR to /opt/drush/8 and adjusting the other commands above accordingly.

Use the same commands with a new version number to update Drush as needed; the commands above account for updates.

Install and update Drush Backdrop commands

In order for Drush to work with Backdrop, you must install the Drush Backdrop commands within the Drush 8 installation itself.

Assuming that Drush 8 is installed in /opt/drush/8/drush/drush, as specified above:

sudo git clone https://github.com/backdrop-contrib/drush.git /opt/drush/8/drush/drush/commands/backdrop

To update Drush Backdrop commands installed like this:

cd /opt/drush/8/drush/drush/commands/backdrop
sudo git pull origin

Install Drush on May First Movement Technology and other shared hosting environments

Primarily drawn from this MF ticket; thanks Ivan!

First, install Composer following these instructions.

Then to install Drush:

echo 'export PATH="$HOME/.composer/vendor/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
cd ~/<organization.org>/bin
composer global require drush/drush:8.4.10
which drush

Note that sometimes cgi-bin is present on MF/PL sites instead of bin.

If using Drush aliases, you may need to specify the new Drush installation in the shared aliases.drushrc.php file or a local aliases file. Here's an example:

$aliases['organization'] = array(
  'remote-host' => 'organization.org',
  'remote-user' => 'organization',
  'root' => '/home/members/organization/sites/organization.org/web',
  'uri' => 'organization.org',
  'path-aliases' => array(
    '%drush-script' => '/home/members/organization/sites/organization.org/users/organization/.composer/vendor/bin/drush',
    '%dump-dir' => '~/drush-backups',
    '%files' => '/home/members/organization/sites/organization.org/web/sites/default/files',),
);

Installing Icinga on Debian server

NEW WAY

Install Icinga from package

aptitude install icinga

Say yes

y

Configuring icinga-cgi

Select apache2

Ok

create icingaadmin password, enter it twice

IT'S INSTALLED!

Make it right!

Change the hostname in the configs

cd /etc/icinga/objects
vim localhost_icinga.cfg

change all instances of "localhost" to be a more useful name for this server, for this example, dev1

:%s/localhost/dev1/g
:x

Do the same for hostgroups_icinga.cfg

vim hostgroups_icinga.cfg
:%s/localhost/dev1/g
:x

restart and see how it looks!

/etc/init.d/icinga restart

In a browser go to serverurl/icinga and log in with the username "icingaadmin" and the password that you set earlier.

External Commands

Icinga is not configured to look for external commands in the default configuration as a security feature. To enable external commands, you need to allow the web server write access to the nagios command pipe. the simplest way of doing this is to set check_external_commands=1 in your Icinga configuration, and then change the permissions in a way which will be maintained across package upgrades (otherwise dpkg will overwrite your permission changes). The following is the recommended approach:

activate external command checks in the Icinga configuration. this can be done by setting check_external_commands=1 in the file /etc/icinga/icinga.cfg.

perform the following commands to change directory permissions and to make the changes permanent:

 service icinga stop
 dpkg-statoverride --update --add nagios www-data 2710 /var/lib/icinga/rw
 dpkg-statoverride --update --add nagios nagios 751 /var/lib/icinga
 service icinga start

Make it report in!

I've put all the files we're going to need into /home/jamila/icingafiles on palante2 to make it a bit easier.

The files are:

 43957 Feb  6 20:51 send_nsca
  1636 Feb  6 20:52 send_nsca.cfg
   997 Feb  6 20:53 submit_check_result
   178 Feb  6 20:57 add_to_commands.cfg

Copy all of those files to some one place on the new server

Move send_nsca and submit_check_result to /usr/sbin/

Move send_nsca.cfg to /etc/icinga

Add the lines in add_to_commands.cfg to the end of /etc/icinga/commands.cfg

If libmcrypt isn't installed, we need to install it.

aptitude install libmcrypt4

Then we need to edit /etc/icinga/icinga.cfg and make the following lines have the following values:

enable_notifications=0

obsess_over_services=1

ocsp_command=submit_check_result

Check the icinga config files to make sure we didn't break anything

/etc/init.d/icinga checkconfig

If all is well, restart!

/etc/init.d/icinga restart

I like to alter the localhost_icinga.cfg to have a different parameters for disk check:

# Define a service to check the disk space of the root partition
# on the local machine.  Warning if < 10% free, critical if
# < 5% free space on partition.

define service{
        use                             generic-service         ; Name of service template to use
        host_name                       alpdebian
        service_description             Disk Space
        check_command                   check_all_disks!10%!5%
        }

If that all works, modify the localhost_icinga.cfg to monitor what you want, and then add the host in palante2 Icinga with the correct service check names.

submit_check_result for this contains:

#!/bin/sh

# Arguments:
#  $1 = host_name (Short name of host that the service is
#       associated with)
#  $2 = svc_description (Description of the service)
#  $3 = state_string (A string representing the status of
#       the given service - "OK", "WARNING", "CRITICAL" 
#       or "UNKNOWN")
#  $4 = plugin_output (A text string that should be used
#       as the plugin output for the service checks)
#

# Convert the state string to the corresponding return code
return_code=-1

case "$3" in
    OK)
        return_code=0
        ;;
    WARNING)
        return_code=1
        ;;
    CRITICAL)
        return_code=2
        ;;
    UNKNOWN)
        return_code=-1
        ;;
esac

# pipe the service check info into the send_nsca program, which
# in turn transmits the data to the nsca daemon on the central
# monitoring server

/usr/bin/printf "%s\t%s\t%s\t%s\n" "$1" "$2" "$return_code" "$4" | /usr/sbin/send_nsca -H icinga.palantetech.com -c /etc/icinga/send_nsca.cfg

add_to_commands.cfg contains

 define command{
        command_name    submit_check_result
        command_line    /usr/sbin/submit_check_result $HOSTNAME$ '$SERVICEDESC$' $SERVICESTATE$ '$SERVICEOUTPUT$'
 }

send_nsca is a binary file, and send_nsca.cfg is our authentication method and password.

Lm-sensors

After installing Icinga, you can set up lm-sensors

OLD WAY

Install Icinga
http://docs.icinga.org/latest/en/quickstart-idoutils.html

You'll need to download Icinga and Nagios plugins to /usr/src

wget http://sourceforge.net/projects/icinga/files/icinga/1.6.1/icinga-1.6.1.tar.gz/
wget http://sourceforge.net/projects/nagiosplug/files/nagiosplug/1.4.15/nagios-plugins-1.4.15.tar.gz

Then copy the send_nsca and send_nsca.cnf from one of the other MSP servers to this server
scp <Icinga server>:/usr/local/icinga/bin/send_nsca /usr/local/icinga/bin/
scp <Icinga server>:/usr/local/icinga/etc/send_nsca.cfg /usr/local/icinga/etc/

Set up a OSCP command as detailed here:
http://docs.icinga.org/latest/en/distributed.html

You can copy the lines in icinga/etc/objects/command.cfg, and the submit_check_result script in icinga/libexec from another MSP server.

old docs here:
https://hq.palantetech.com/issues/1582
https://hq.palantetech.com/documents/120


Instrucciones para clientes web y de escritorio ownCloud

Su organización ahora está utilizando ownCloud para para compartir archivos. ownCloud es software como Dropbox, pero en vez de funcionando en un servidor de otra compañía, el software funciona en un servidor que es propiedad de su organización. Con esto, ustedes tienen más control y seguridad.

Acceso al ownCloud

Para acceder a ownCloud, abra un navegador web y vaya al url que te dimos. Su nombre de usuario es su nombre de pila; el url para cambiar su contraseña llegará vía email.

Navegando por ownCloud

Cuando su sesión es iniciado, debería ver el contenido de sus carpetas compartidas. Para navegar entre carpetas, haga clic en el nombre (no en el icono) de la carpeta que desea entrar. En la parte superior de la pantalla, puede ver la ruta del archivo de sus subcarpetas. Puede volver a subir otra vez haciendo clic en el icono de inicio (la casa) o cualquier otra carpeta en la ruta del archivo.

Descargando y renombrando archivos

Para descargar un archivo, haga clic en el icono de la elipsis ("...") junto al archivo. El menú que aparece te permitirá descargar o renombrar archivos. También puede renombrar o descargar carpetas; al descargar una carpeta, va a ser comprimida antes de descargar, por lo tanto debe ser cómodo con la descompresión si decide hacer esto. Descargando un archivo grande se llevará mucho tiempo, y una carpeta que es demasiado grande no se pueden descargar a la vez.

IMPORTANTE: ¡Nunca eliminar archivos! Si elimina accidentalmente un archivo, haga clic en el enlace "Archivos eliminados" en la esquina inferior izquierda, entonces haga clic en el icono de la elipsis junto al archivo y haga clic en "Recuperar" para recuperar el archivo en su ubicación original.

Cargando archivos

Para cargar o subir un archivo, haga clic en el icono signo más ("+") en la parte superior de la pantalla y seleccione "Subir." También puede dragear un archivo o una carpeta de su computadora a la pantalla del navegador web para subirlo.

Compartiendo archivos

Haga clic en el icono de compartir para abrir un panel de opciones de compartir a la derecha de la pantalla. Desde aquí, haga clic en "Enlace compartido" para recibir un enlace que puedes compartir por email con colegas fuera de su organización por lo que puede descargar el archivo. Esto es preferible a enviar un archivo directamente como un archivo adjunto de email porque los archivos pueden ser de tamaño ilimitado, y si el archivo se actualiza en ownCloud, se descargue la versión más reciente del archivo cuando utiliza el enlace que enviaste. También puede hacer protección con contraseña o establecer fecha de caducidad.


Updated almost 4 years ago by Janine Ko

Welcome to Owncloud

Your organization is now using ownCloud for file sharing. ownCloud is software similar to Dropbox, Google Drive, or SkyDrive, but instead of running on a third party's server, it runs on a server in your office. This allows us to combine features of Dropbox with your in-house shared drive seamlessly. OwnCloud has two interfaces for users to interact with the files stored there--the web portal and the desktop applications.

ownCloud's web portal

Accessing ownCloud

To access ownCloud, open a web browser and go to the site listed below for your organization. Enter the same username and password you use to log into your office computer in the morning.

A.J. Muste: https://cloud.ajmuste.org/owncloud
ALP: https://cloud.alp.org/
CPR: https://cloud.changethenypd.org
GPP: https://share.mayfirst.org
Legal Momentum: https://emma.legalmomentum.org/gdrive
NLG: https://nlgserve.nlg.org/
RFF: https://cloud.rainforestfoundation.org/owncloud/
SRLP: https://office.srlp.org/owncloud

Navigating ownCloud

Once you're logged in, you should see the contents of your shared drive. To navigate between folders, click on the name of a folder (not the icon of the folder) to enter the folder. Along the top of the screen, you'll see the "path" of your subfolders - you can go back up by pressing the "Home" icon, or any other folder in the path.

Downloading, renaming, deleting files

To download a file, click the ellipses icon ("...") next to the folder. The menu that appears will allow you to download, rename or delete a file. It's also possible to download folders this way; they will be zipped up before download, so you should be comfortable with unzipping if you choose to do this. Note that downloading a large file will take a long time, and a folder that's too large can not be downloaded at once.

Uploading files

To upload a file, press the plus button ("+") at the top of the window and select "Upload". You can send files from your computer to the server. You can also drag a file or folder from your computer onto the web browser window to upload it.

Sharing files

If you click the sharing icon next to a file, you'll bring up a "sharing" panel on the right. From here, you can click "Share Link" and receive a link you can e-mail to colleagues outside the organization allowing them to download the file. This is preferable to e-mail because the files can be of unlimited size, and if you update the file on the shared drive, they'll get a copy of the latest version. You can also password-protect the link and/or set it to expire after a set number of days.

Cheatsheet

Below please find an annotated screenshot of all the options you may want to use in ownCloud.

ownCloud's desktop client

Installation


Introduction to G Suite

Welcome to G Suite! We've created a Google account for you, which includes a suite of different tools including:

Your organization now manages all users and information under accounts ending with <yourdomainname>.org. You can login and view your account information at: http://accounts.google.com/

Gmail

Your email can be accessed here: http://gmail.com/

Google has created a pretty comprehensive guide to Gmail's web functionality. We'll be walking through these parts in the webinar.

Google also has a guide to its web interface that is specific to former Outlook users. that you can consult if you're looking to replicate certain Outlook functionality in the web interface.

Google Calendar

Your calendars can be accessed here: https://calendar.google.com/

Google's made this comprehensive guide to Calendars. We'll be walking through these parts in the webinar.

There are some differences between Outlook and Google Calendars.

Google Drive

You can access your Google Drive here: https://drive.google.com/

Google's guide to Google Drive is here. We'll be walking through these parts of it in the webinar.

Setting up Google Drive Filestream


Updated over 8 years ago by Jon Goldberg

Introduction to ownCloud

Hi there,

I've set up an account for you on ownCloud. ownCloud is similar to Dropbox or Google Drive, which we don't use out of a concern for keeping our clients' data private and secure. Our ownCloud is maintained by May First/People Link, an organization with a strong history of defending the privacy of its members' online data.

To log in, go here:
https://share.mayfirst.org

Your username is: <username>
Your password is: <password>

We very strongly recommend changing this password before your first use. To do so, please go to:
https://share.mayfirst.org/cp

Please upload documents to me by clicking on the <shared folder> folder, then dragging the files onto your web browser. I may also ask you to download files from the same location. If you have sensitive data (such as a password or client data) please do not send it via e-mail; please upload it via ownCloud, then notify me via e-mail.

<client specific instructions here, including which folder to download/upload into>

Use the same username and password as above. The first screen after login will allow you to change your password.

Thanks!
Jon


Introduction to ownCloud

Your organization is now using ownCloud for file sharing. ownCloud is software similar to Dropbox, Google Drive, or SkyDrive, but instead of running on a third party's server, it runs on a server belonging to NDWA. This allows NDWA greater control and security.

Accessing ownCloud

To access ownCloud, open a web browser and go to the site listed below. Your username is your first name; a password reset link should have been sent to you by email.

NDWA: https://cloud.domesticworkers.org/owncloud

Navigating ownCloud

Once you're logged in, you should see the contents of your shared folders. To navigate between folders, click on the name (not the icon) of the folder you'd like to enter. Along the top of the screen, you'll see the "path" of your subfolders - you can go back up by pressing the "Home" icon, or any other folder in the path.

Downloading, renaming, deleting files

To download a file, click the ellipsis icon ("...") next to the folder. The menu that appears will allow you to download, rename or delete a file. It's also possible to download folders this way; they will be zipped up before download, so you should be comfortable with unzipping if you choose to do this. Note that downloading a large file will take a long time, and a folder that's too large cannot be downloaded at once.

Uploading files

To upload a file, press the plus button ("+") at the top of the window and select "Upload". You can send files from your computer to the server. You can also drag a file or folder from your computer onto the web browser window to upload it.

Sharing files

If you click the sharing icon next to a file, you'll bring up a "sharing" panel on the right. From here, you can click "Share Link" and receive a link you can e-mail to colleagues outside the organization allowing them to download the file. This is preferable to e-mail because the files can be of unlimited size, and if you update the file on the shared drive, they'll get a copy of the latest version. You can also password-protect the link and/or set it to expire after a set number of days.

Cheatsheet

Below please find an annotated screenshot of all the options you may want to use in ownCloud.


Last updated by Jon Goldberg Updated about 8 years ago

Introduction to ownCloud desktop - in-house


Updated 10 days ago by Angel Pleasant

IT Managed Service changelog


K9 Setup


Updated almost 2 years ago by Jack Aponte

Maintenance plan changelog


Mapping Polygons and Countries

The basic setup and creation of maps follow the same basic 3-steps as outlined in the Basic Mapping Documentation. Mapping Polygons simply requires selecting the correct Geofield widget and understanding the UI. Mapping countries, however, takes installing two new modules, using a new field type and setting up the data layer from step 1 slightly differently.

Mapping Polygons

The setup for polygons follows the same basic data setup as points: Create a content type for the data we're collecting, and add a geofield. Select OpenStreet Map as the widget. Set the field to be required and adjust the default map center and zoom for data entry if that's needed. The rest of the defaults should be fine to accept. When editing the content, the map appears for the user to enter the data.

To create an area on the map instead of a point, select the polygon tool in the top right hand corner of the map. It looks like a closed set of lines that's shaded in the middle. Then click you're first point on the map to start the polygon. You must select them in a single direction around the outside of the shape. You can't add points between two already added ones. To finish selecting, hold down the shift key when selecting the last one to release the drawing. There's a polygon on the map widget that will render as a series of longitude and latitude points when viewing the content.

Creating a map of polygon data follows the same three steps as described in the Basic Mapping Documentation. The data formatter when creating the aggregation layer is the same as a single point, WKT. Hand creating polygons is an unlikely data entry scenario, though. It is demonstrative for using a series of longitude and latitude like for mapping countries.

Mapping Countries

Since country borders are unstable, maintaining the exact longitude and latitude could become cumbersome. As such there's a countries border module we can install that maintains these points for us. We simply select the country when entering the data, and reference that selection when building the map with the actual coordinates. If the borders of a country change, the module will be updated and subsequently our maps will be automatically updated.

We first have to install two modules, countries and countries_borders. The former provides an explicit field data type for the country, while the latter provides the longitude and latitude coordinates for each. The former is for data entry; the latter for data aggregation.

Create a content type and add a field type of country and choose the widget of Countries by Continent. This is a chain selected list of countries, which is better data entry presentation than the entire list of countries as radio buttons or checkboxes or the error prone open text box. Though these all do have their uses. In some cases it might be desired to list all the available countries of a given content type, and since it's possible to limit the available in the next configuration step, we could list all the available countries. Set the field to be required and unless you're limiting the available countries there are no other configuration options needed. When creating or editing the content, only the countries are listed. There is no direct mapping data. That's handled at the data aggregation step, step 1 of the normal 3 step process for creating maps.

After creating the data aggregation view and before adding any fields, you need to add a Relationship to the view. Click open the Advanced section on the right hand side, click Add next to the Relationship heading. Then select the content country field we created in the above steps. The defaults should be fine; we don't need to require the relationship.

Now we proceed to adding the fields. We add a Country: Borders field, which is the mapping data from the countries_border module. We configure this field to use the relationship we created to the country data from the content type; it should be the only option. Formatter should be WKT and data options should be Use full geometry. Conceptually the way this field work is it will provide the longitude and latitude coordinate series for what country we've selected when entering the data. That's how the relationship works. Add the Title and the Body as we've done with other maps in order to provide information about the data being mapped.

When configuring the OpenLayers Data Overlay, set the Map Data Sources to WKT and select the Country: Borders field as WKT source. The final steps for mapping the countries will follow the same steps as in the Basic Mapping Documentation.


Mapping Solutions

Open Source Mapping fundamentals

Both are a part of the OSGeo supported suite, which is worth investigating for other solutions.

Mapping application suites

Data collection apps primarily geared toward humanitarian efforts


Migrating CiviCRM to a new site

This page is deprecated

I've merged this page into Moving an Existing Installation to a New Server or Location on the CiviCRM wiki. I've also referenced it from the Migrating from Drupal to Wordpress and Moving an Existing CiviCRM Installation from one Drupal frontend to another pages.


Monitoring

Installing Icinga on Debian server
Setting up NRPE on Debian server
Icinga response procedure


Updated almost 5 years ago by Morgan Robinson

On Call Troubleshooting

The on-call system functions basically as a first-tier of support: clients see more progress sooner, and tickets can be handled with a clearer understanding after being processed once by a worker. This is a list of troubleshooting tips to clarify that first pass. This does not supersede the on-call protocol (i.e. if unclear, ask if an issue is urgent). When a ticket comes in, claim it for yourself and acknowledge it (e.g. "Thanks for getting in touch with us! I'm going to do some initial research right now to assist in troubleshooting, and will report my findings when I escalate the ticket."). This process should involve about 15 minutes of research, after which the ticket should be passed along with your findings.

Any time a ticket comes in, the questions are:

Updated over 3 years ago by Janine Ko

Opening Another Mailbox in OWA

Complete instructions here: https://support.microsoft.com/en-us/office/open-and-use-a-shared-mailbox-in-outlook-web-app-bc127866-42be-4de7-92ae-1ef2f787fd5c

  1. Sign in to your account in Outlook Web App.
  2. On the Outlook Web App navigation bar, select your name. A list appears.
  3. Select Open another mailbox.
  4. Type the email address of the other mailbox that you want to open and then select Open. Or, start typing and then select Search contacts and directory to find the mailbox you want to open. Select the shared mailbox you want to open, and then select Open. Another Outlook Web App session opens in a separate window, allowing access to the other mailbox.

Handling CiviCRM on dev/staging sites

Moving the CiviCRM database

CiviCRM database dumps contain DEFINER statements that specify a mysql user for the purpose of creating triggers. If the database is loaded from live with an incorrect user, the dev/staging site may not work correctly and backups will not run.

You can also remove the DEFINER lines using. Although you want to save the original

$: sed 's/\sDEFINER=[^`]*`@`[^`]*//g' -i sql_dumpfile.sql 

Resource URL overrides

In development and staging environments, you can override CiviCRM options on dev/staging sites in civicrm.settings.php. Once overridden in this file, any resource URLs or other settings stored in the database will not disrupt the site when the database is synced from the live site.

Example settings for paths below. These lines may go at the top of civicrm.settings.php just under the opening PHP code tag, so they are obvious to other users.

global $civicrm_setting;
$civicrm_setting['Directory Preferences']['customTemplateDir'] = '/var/www/mysite/sites/all/civicrm/templates';
$civicrm_setting['Directory Preferences']['customPHPPathDir'] = '/var/www/mysite/sites/all/civicrm';
$civicrm_setting['Directory Preferences']['extensionsDir'] = '/var/www/mysite/sites/all/civicrm/extensions';
$civicrm_setting['URL Preferences']['extensionsURL'] = 'http://mysite.local/sites/all/civicrm/extensions/';
$civicrm_setting['URL Preferences']['imageUploadURL'] = 'http://mysite.local/sites/default/files/civicrm/persist/contribute/';
$civicrm_setting['URL Preferences']['userFrameworkResourceURL'] = 'http://mysite.local/sites/all/modules/contrib/civicrm';

It's also possible to override other values in the Setting object in this file to facilitate development or testing.


Updated almost 8 years ago by Jessie Lee

Owncloud

Owncloud has issues with .DS_Store files. At first I thought it was a Dropbox <-> Owncloud error, but then I turned off my Dropbox and was still getting the error. We need to figure out a way to have Owncloud exclude syncing .DS_Store files, or else it will generate a bunch of them. I don't know why it is doing so, but I have figured out how to delete them all:

Run this in the top of the Owncloud directory.

find ./ -type f -name ".DS_Stor*" -exec rm {} \;

Owncloud Installation

Owncloud 9.0 on Debian 8 with Apache

Apache setup

Database setup (mySQL)

First Time Wizard

Enabling Samba external users

Samba External Storage

owncloud + nginx

Performance tweaks

Security and Hardening

For Office File servers

Owncloud does not autoscan files that are not controlled by owncloud regularly. For office file servers this could mean files added via different methods may not show up in owncloud.

Owncloud may also need the web user (debian www-data) to be part of the staff group. usermod -a -G www-data user

To add users /home folders:

Fixing Permissions

resource links


Last updated by Jon Goldberg Updated about 8 years ago

OwnCloud Administration

User Administration on May First

If you don't see the answer to your question here, please be sure to check these other resources:
ownCloud documentation site: https://doc.owncloud.org/
May First ownCloud documentation: https://support.mayfirst.org/wiki/owncloud

Adding/removing users

If your ownCloud host is May First (https://share.mayfirst.org), user administration must be done through the May First control panel.

To add/remove users from the system, log into https://members.mayfirst.org/cp with an administrative user. Please contact us if you're not sure which users are administrative users.

To add users, click the "Add Item" button on the top. Fill in the first name/last name/username/password (or random generate password). Creating a user in May First isn't enough to share a document with someone; they also need to log into ownCloud at least once. You may want to log in for the first time for the user.

Changing passwords

The instructions above for adding/removing users can also be used for changing passwords.

Adding/removing administrative users

Adding users to groups

To add a user to a group, they must first be in a group you already have "Group Admin" privileges to. For instance, GPP staff would be in a group called "GPP All", and from there they can be added to other groups owned by the GPP user.

To add a user to that first group, you must currently submit a ticket to May First support. Go to https://support.mayfirst.org, log in using the same username/password you use for ownCloud, and click "New Ticket" at the top. Be sure to include the name of your organization, the group you'd like the users added to, and their exact usernames.


Updated over 4 years ago by Jamila Khan

Owncloud cli cheatsheet

Architecture

OCC

Occ is the cli command tool for owncloud. Through this tool most user/file/maintenance operations can be performed.
detailed instructions can be found here https://doc.owncloud.org/server/9.0/admin_manual/configuration_server/occ_command.html

key notes/favorite commands:

Command line upgade procedure

Docker upgrade procedure

Old Upgrade Procedure


Using the ownCloud web client

Your organization is now using ownCloud for file sharing. ownCloud is software similar to Dropbox, Google Drive, or SkyDrive, but instead of running on a third party's server, it runs on a server belonging to NDWA. This allows NDWA greater control and security.

Accessing ownCloud

To access ownCloud, open a web browser and go to the site listed below. Your username is your first name; a password reset link should have been sent to you by email.

NDWA: https://cloud.domesticworkers.org/owncloud

Navigating ownCloud

Once you're logged in, you should see the contents of your shared folders. To navigate between folders, click on the name (not the icon) of the folder you'd like to enter. Along the top of the screen, you'll see the "path" of your subfolders - you can go back up by pressing the "Home" icon, or any other folder in the path.

Downloading, renaming, deleting files

To download a file, click the ellipsis icon ("...") next to the folder. The menu that appears will allow you to download, rename or delete a file. It's also possible to download folders this way; they will be zipped up before download, so you should be comfortable with unzipping if you choose to do this. Note that downloading a large file will take a long time, and a folder that's too large cannot be downloaded at once.

Uploading files

To upload a file, press the plus button ("+") at the top of the window and select "Upload". You can send files from your computer to the server. You can also drag a file or folder from your computer onto the web browser window to upload it.

Sharing files

If you click the sharing icon next to a file, you'll bring up a "sharing" panel on the right. From here, you can click "Share Link" and receive a link you can e-mail to colleagues outside the organization allowing them to download the file. This is preferable to e-mail because the files can be of unlimited size, and if you update the file on the shared drive, they'll get a copy of the latest version. You can also password-protect the link and/or set it to expire after a set number of days.

Cheatsheet

Below please find an annotated screenshot of all the options you may want to use in ownCloud.

Installing the ownCloud Desktop client


Owncloud Office Share

Madre

Firefox:

Owncloud main screen

Desktop Client Setup


Payment Processor Test Cards

Authorize.net

Visa 4111 1111 1111 1111
https://developer.authorize.net/hello_world/testing_guide/

PayJunction

Visa 4242 4242 4242 4242
https://developer.payjunction.com/hc/en-us/articles/216477397-Development-Sandbox

Stripe

Visa 4242 4242 4242 4242
https://stripe.com/docs/testing#cards


Updated over 5 years ago by Jon Goldberg

PayPal and CiviCRM

PayPal is very difficult to navigate, and very difficult to handle certain tasks that come up time and again, particularly around recurring payments. I'm documenting them here for future reference.

General CiviCRM documentation

http://wiki.civicrm.org/confluence/display/CRMDOC/PayPal+Website+Payments+Standard+and+Recurring+Contributions
http://wiki.civicrm.org/confluence/display/CRMDOC/PayPal+Website+Payments+Pro+and+Express+Configuration

Setting a URL for IPN callbacks

Go here: https://www.paypal.com/us/cgi-bin/?cmd=_profile-ipn-notify

Resending an IPN

Do this from the IPN History page: https://www.paypal.com/us/cgi-bin/webscr?cmd=_display-ipns-history. Note the only way to find this page is from the "Setting the IPN callback" link above.

Modifying an IPN to reflect an updated CiviCRM contact ID, etc.

This is my raw documentation for the most complicated version of this. In this case, a user had a recurring membership/contribution, but a database restore eliminated all references in CiviCRM to it. However, the recurring payment profile still existed in PayPal.

I built a new recurring contribution from scratch. I compared an existing recurring contribution1 and then used the values from an IPN that corresponded to Carolyn2
[1] select * from civicrm_contribution_recur WHERE id = 700;
[2] select * from civicrm_system_log WHERE id = 223;

PayPal:
My Account > Profile
Recurring Payments Dashboard (https://www.paypal.com/us/cgi-bin/webscr?cmd=_merchant-hub)
Scrolled back to the creation date and clicked on Carolyn's name
Clicked "Modify".
The profile reference has a bogus maximum field length, so I right-clicked, selected "Inspect Element", and manually changed the "maxlength" value so I could edit it.

Finally, I created a new membership through the UI (46913) and edited THAT manually to have updated end date. I also created a membership_payment record.


Updated almost 9 years ago by Jessie Lee

PDF to Image Using PDFcreator

PDFcreator can be downloaded here: http://www.pdfforge.org/pdfcreator/download
version 1.7.3 is used in this example.


Updated about 9 years ago by Jon Goldberg

Prefix/Suffix Mapping

Here, we're referring to name prefixes like "Mr.", "Ms.", "Dr.", "Senator", and suffixes like "Esq.", "Sr.", "MD", and so on.

When editing an individual's prefix/suffix, CiviCRM requires you to pick from a list. It's easy to edit, but is intended to avoid the common situation of seeing "Mr", "Mr.", "MR.", and "mr." all in the same database. When I migrate your data from another system, I can automatically remap your prefixes/suffixes to standardize them, drop bogus prefixes/suffixes, and add unusual but legit prefixes/suffixes to your CiviCRM list.

You should have in your ownCloud folder (or an e-mail from me) two files - one called "invalid prefixes.csv" and the other "invalid suffixes.csv". Please open them in a spreadsheet program (Excel, LibreOffice Calc, Google Spreadsheets, etc.) and place the following in column B:

Please see the remapping below for inspiration on standardizing!

Source value Target value
Bro. Brother
Br. Brother
Mr. Dean Dean
DR Dr.
DR. Dr.
Doctor Dr.
Dr Dr.
dr Dr.
Father Fr.
Father Fr.
Fr Fr.
Judge The Honorable

Special note on Salsa

If you're migrating from Salsa, you'll see a lot of job titles in the "Prefix" field. This is because in Salsa, this field is called "Title", which is confusing to folks. Unfortunately, since the field is intended for prefixes, it's limited to 16 characters - so there's not even any sense in trying to move that info into a job title field. Just blank those!


Resending lost messages to Redmine

When Redmine goes down, emails sent to Redmine don't get automatically resent.

Messages sent to tickets@palantetech.coop are both redirected to tickets@maple.palantetech.coop, but also a copy is kept on the albizu server for 30 days. To resend messages that never made it to maple, one must connect to albizu.

Using Thunderbird (easiest):

Set up an IMAP account for tickets@palantetech.coop. SMTP/IMAP server is mail.mayfirst.org. Username is "tickets", password is either in creds, or reset it from the MFPL control panel.
You must have the Mail Redirect extension installed.
Select the messages you want to resend in the mailbox, click "Redirect", and enter the e-mail address of "tickets@maple.palantetech.coop".

Using mutt

mutt doesn't require any setup, but can only redirect one message at a time. In theory you could speed this along by making a macro in muttrc, but that's beyond what we need IMO.

ssh tickets@albizu.mayfirst.org
mutt
:set sort = reverse-date-received
1 #go to first message
b #for "bounce", aka redirect
tickets@maple.palantetech.coop
# alternatively, send to tickets-test@maple.palantetech.coop if need be

Updated almost 8 years ago by Jessie Lee

ResourceSpace Installation

Initial setup

Database (mysql)

Apache config

PhP requirements

First Run Wizard

Cron

SMTP

SSL setup


Set up local bounce processing

If you're setting up CiviMail, you'll need a bounce processing account on a system that supports subaddressing. GMail/Google Apps works for this - Exchange and Office 365 do not. So it's often desirable to set up your own server for bounce processing - it takes very little overhead, so you can run it on the same VPS that runs CiviCRM.


Setting up NRPE on Debian server

Process:

on client server

Install NRPE

apt install nagios-nrpe-server nagios-plugins-basic

Edit config file

vim /etc/nagios/nrpe.cfg

start service

service nagios-nrpe-server restart

check that service is running

netstat -tpln | grep 5666

edit local config to add specified checks

vim /etc/nagios/nrpe_local.cfg

Add NRPE test check, disk check, and Apache SIGTERM check

0 octavia:/etc/nagios# cat nrpe_local.cfg 
######################################
# Do any local nrpe configuration here
######################################

command[check_nrpe_daemon]=/bin/echo "NRPE OK" 

# disk checks
command[check_disk_root]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/root

# checking Apache logs for SIGTERM
command[check_apache_sigterm]=/usr/lib/nagios/plugins/check_cat.sh /var/log/apache2/error.log SIGTERM 0

add check_cat

In /usr/lib/nagios/plugins create check_cat.sh

#!/bin/bash

#
# checks in files if there are instances of a given string.
# $1 is the file
# $2 is the string
# $3 is the threshold, any more than $3 and it will be Critical

cnt=`cat $1|grep $2|wc -l`
recent=`cat $1|grep $2`
if [ $cnt -le $3 ] ; then
        echo OK - no errors or warnings
        exit 0
fi

echo -e "CRITICAL - String $2 appeared $cnt times\n$recent" 
exit 2

Make it executable

chmod 755 /usr/lib/nagios/plugins/check_cat.sh

Make total procs check ignore kernel processes

Add -k to the total_procs check in /etc/nagios/nrpe.cfg, and lower the threshold.

command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -k -w 130 -c 180

restart service

service nagios-nrpe-server restart

on monitoring server

test connection

On Nagios/Icinga server test that that worked:

/usr/lib/nagios/plugins/check_nrpe -H clientserveruri.com -c check_nrpe_daemon
NRPE OK

set up checks.

example service config file

###############################################################################
###############################################################################
#
# SERVICE DEFINITIONS
#
###############################################################################
###############################################################################

define service{
        use                     generic-service         ; Inherit default values from a template
        host_name               clientserver
        service_description     SSH
        check_command           check_ssh
        }

define service{
        use                     generic-service         ; Inherit default values from a template
        host_name               clientserver
        service_description     HTTP
        check_command           check_http
        }

define service{
        use                     generic-service         ; Inherit default values from a template
        host_name               clientserver
        service_description     Users
        check_command           check_nrpe_1arg!check_users
        }

define service{
        use                     generic-service         ; Inherit default values from a template
        host_name               clientserver
        service_description     Load
        check_command           check_nrpe_1arg!check_load
        }

define service{
        use                     generic-service         ; Inherit default values from a template
        host_name               clientserver
        service_description     Zombie Processes
        check_command           check_nrpe_1arg!check_zombie_procs
        }

define service{
        use                     generic-service         ; Inherit default values from a template
        host_name               clientserver
        service_description     Total Processes
        check_command           check_nrpe_1arg!check_total_procs
        }

define service{
        use                     generic-service         ; Inherit default values from a template
        host_name               clientserver
        service_description     Disk Space /root
        check_command           check_nrpe_1arg!check_disk_root
        }

define service{
        use                     generic-service         ; Inherit default values from a template
        host_name               clientserver
        service_description     Apache SIGTERM
        check_command           check_nrpe_1arg!check_apache_sigterm
        }

restart monitoring service

for icinga

/etc/init.d/icinga restart

sources:
http://xmodulo.com/2014/03/nagios-remote-plugin-executor-nrpe-linux.html
https://wiki.icinga.org/display/howtos/Setting+up+NRPE+with+Icinga


Updated over 7 years ago by Jon Goldberg

Shared Drop

To share documents of any size with folks at Palante Tech, you can upload them to our shared drop. This is similar to Dropbox or Box.net, but more secure and private.

Palante staff instructions


Updated about 9 years ago by Jon Goldberg

Shared folders for migration

You should now have a folder named "civicrm" shared with you via ownCloud. ownCloud is software similar to Dropbox or box.net, but it's open source software - so we're able to store our files on privacy-conscious secure hosting. You can access them either through the web or by installing the ownCloud client. Most likely, you'll want to use the web interface.

Inside your shared folder "civicrm" will be four (or more) subfolders. The most important folders are:

Updated about 9 years ago by Jon Goldberg

Shared Webhost options

May First/People Link, Koumbit, and Electric Embers are politically conscious hosts with a commitment to protecting their clients' data.

May First/People Link, $200 per year ($16.67 per month)
https://mayfirst.org/hosting

Koumbit, Package A, $100 per year ($8.33 per month)
http://www.koumbit.org/en/services/hosting

Electric Embers (coop), the Private option. Pricing is based on a
sliding scale but will likely be $25 per month.

Gaia Hosting, Standard package, $15.00 per month + $15.00 setup fee.
http://www.gaiahost.coop/services/info/5-web-application-standard-email-hosting

Green Geeks, $4.95 per month
http://www.greengeeks.com/web-hosting/

Liquid Web, Standard Plan, $14.95 per month.
https://www.liquidweb.com/configure/standard-website-hosting-plan


Statement on Union Solidarity

Palante stands in solidarity with non-profit employees unionizing across the country, exercising their right to collectively bargain for better wages, benefits, and working conditions. As a democratic, worker-owned cooperative that serves non-profit and community organizations advancing social justice, we believe that the internal structures of our movements and organizations should model the liberated world we are trying to build. Unions are critical to protecting worker dignity and power in a sector where workers who do the life-sustaining work of community outreach and support on the ground are often compensated with the lowest wages and meager benefits. Unions help equalize the power relationship between employers and employees, giving workers leverage to demand greater organizational transparency and recourse in cases of impunity. We believe that unionization, a right guaranteed by law, builds more equitable organizations and ultimately stronger movements.

Palante reserves the right to terminate our contracts with any of our clients if we discover the use of union-busting tactics by management. We stand beside workers who strive to hold non-profit organizations accountable and to transform their workplaces for the better.


Tech client move checklist

Questions to answer as far in advance as possible while assessing potential new spaces

1-3 months before move

1 month before move

2-3 weeks before move

1 week before move

Move days


Tools


Topics for "CiviCRM 102" training

Let's also make sure we know what we're covering! Here's the topics I intend to cover:

Unhacking a WordPress site

This is an INCOMPLETE guide, but a good starting point! It doesn't cover removing malicious code inserted into the database, for instance.

WordPress gets hacked - a lot. And the correct solution is to restore your database and filesystem from backup. However, sometimes we deal with sites that weren't responsibly managed, and that's not an option. Here's a guide on what to do.

First - if it IS an option, delete your WordPress filesystem and restore from known good files. There's just too many ways to obfuscate a hack, so these approaches are necessarily incomplete.

Not every instance of these commands is malicious! However, a hacked site will often use these, so look at what comes after them. If it's a long base64 block, that's bad news.

Note that there are MANY ways to obscure the commands above. Here are some example strings you can also search for

"base" . "64_decode" 
eval/*

That last one's tricky. It found this command: eval/*boguscomment*/('malicious_command').

That evaluates to:

<?php eval(get_option("render")); ?>

This indicates that there's malicious code in your database, and this minimal change allows the code to render.

Here's the commands I used to remove that from my entire codebase:

find -name \*php -exec sed -i 's/<?php eval(get_option("\\x72\\x65\\x6e\\x64\\x65\\x72")); ?>//g' {} \;
find -name \*.html -exec sed -i 's/<?php eval(get_option("\\x72\\x65\\x6e\\x64\\x65\\x72")); ?>//g' {} \;

From when Highlander was hacked:

If the site is hacked (note that I wrote these fast and before my vacation... they should probably be updated and my guess is this process can be refined)

  1. Confirm that the site is hacked or being actively attacked
  2. If it being is compromised ssh into the site
  3. go to the web root directory
  4. vim .htaccess
  5. uncomment line 94 and change it with your ip so it should be something like
    Require ip your.ip.goes.here  another.ip.goes.here
  6. save it. and confirm that only the folks from those ips can access the site
  7. Login into the site with Manu's user name, password
  8. goto plugins
  9. activate Wordfence Security and run the malware scan
    1. this may take some time to complete.
    2. there will be some red herrings as well because of files not updated
  10. If you find something confirm that the it's a hacked file...
    1. When you ssh into the root directory, run git status to see if any files have been changed.
    2. If the same file pops up, the site might be compromised.
  11. If the site is indeed compromised. clean up up the bad files.
  12. Check the database for compromised data.
  13. First I check if there were new users created:
      wp user list --role=administrator --format=table 
  14. If there bogus users, you'll need to delete them either through the wp cli or through web admin panel.
  15. Check for bogus post data
  16. If there is bogus data it's probably best to go into updraft plus and revert to the last back-up
  17. you may want to change passwords at this point for users.
  18. If that worked you can change the .htaccess file back by commenting out line 94
  19. confirm that the site is working for all.
  20. monitor for a few days

Use Customs Outlook Templates

Using for email

creating and adding templates to outlook


VPS webhost options

May First/People Link and Koumbit are politically conscious webhosts with a commitment to keeping data safe from government surveillance.

May First/People Link
$100-300/month (sliding scale)
Managed VPSes only
4 gigabytes RAM, 100 gigabytes disk, unlimited transfer

Linode
Price: https://www.linode.com/pricing
VPS management: $100/month

Digital Ocean
Price: https://www.digitalocean.com/pricing/
Unamanaged VPSes only

Koumbit
Price: https://www.koumbit.org/en/services/vps
Managed VPS for additional fee
Note: Prices are in Canadian dollars.


Webmin

Webmin is the web interface for folks to handle day-to-day server tasks without calling Palante Tech for assistance. You typically access Webmin by opening a web browser and going to https://localhost:10000. That link will only work while inside the office!

Creating a new user

From there:

Website Planning Questions

What will this website need to do to meet our goals?

This can be a mix of both short-term and long-term goals. While these should certainly be related to the mission and goals of your organization, website goals tend to be more specific and concrete than the organizational vision.

Examples of website goals include:

Who comes to your site, and why do they come?

Create a list of everyone who might visit your site and whose needs you'd like to meet. For each audience, broadly describe their motivation for coming to the site. Be sure to think about your audiences' goals, rather than the goals of your organization or campaign; that will help you develop a site that's centered on your visitors, effectively engages them, and keeps them interested.

Example audiences include:

What do those audiences want from your site? What do you want from them when they come to the site?

It's important to put your audience's specific website needs first; by meeting visitors' immediate needs, you have a better chance of encouraging them to take the kinds of actions that will help you meet your website goals. It's give and take!

Examples:


Updated about 1 month ago by Angel Pleasant

Welcome to the Palante Technology Cooperative Commons Wiki.

As part of our commitment to transparency and open source principles, we try to make our internal documentation available publicly. We hope that this assists others who may be starting their own cooperatives, looking for best practices on web development, and more.

To contact us: You can reach us by phone or e-mail from here: https://palantetech.coop/contact

Licensing:

Unless otherwise specified, all material with https://redmine.palantetech.coop/projects/commons/wiki/ in the URL is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Cooperative documents

Bylaws - This document is licensed under a CC-0 license.
Certificate of Incorporation - This document is licensed under a CC-0 license.
Conflict Resolution Sheet template
Grievance Policy
Worker Evaluation Questionnaire
Accountability Process

Statements

Statement on Union Solidarity

User documentation

Email

K9 Setup
How to set up Microsoft 365
Opening Another Mailbox in OWA
Intro to G Suite

General Resources

Choosing a webhost
PDF to Image Using PDFcreator
Tech client move checklist

ownCloud

Introduction to ownCloud
Instructions for ownCloud web and desktop clients
Instrucciones para clientes web y de escritorio ownCloud
ownCloud Administration
OwnCloud instructions for NDWA
Shared Drop

Security & Privacy

How to Enable Private Browsing

Technical Documentation

Computers

Windows 10 Wake On Lan fixes

Backups

Borg backups
IBackup for linux

General

Skillshares

Monitoring

Monitoring

Servers maintenance, shares

101 Uses for Curl - WIP
Common Problems
Debian_7_to_8
Debian 8 to 9
Debian 9 to 10
Debian 10 to 11
OwnCloud
Owncloud cli cheatsheet
Resending lost messages to Redmine
ResourceSpace Installation
Upgrades

Web Development and Maintenance

Compass
Development environment
Git
Gitolite
MySQL and Apache Tuning
PayPal
Setting up a new site
Unhacking a WordPress site

Drupal, Backdrop and Web

Drupal and Backdrop Module Matrix
A table of Drupal 7 modules commonly used by Palante when building websites and their status for Drupal 8 and Backdrop
Installing Drush for working with Drupal 6-8 and Backdrop
Website Planning Questions

CiviCRM

CiviCRM user story examples
Prefix/Suffix Mapping
CiviCRM Kettle Transforms
CiviCRM Monitoring Troubleshooting

Mapping

Mapping Solutions
Basic Mapping Documentation
Mapping Polygons and Countries
Custom Mapping Style

Sample Contracts and Changelogs

These documents are licensed under a CC-0 license.

Sample Web/CiviCRM/VPS/NextCloud Maintenance/Support Plan contract
Maintenance plan changelog

Sample IT Managed Services Plan (MSP) contract
IT Managed Service changelog

Sample ADAP v1 contract
Sample ADAP v2 contract
ADAP Hosting Plan changelog (versions 1 and 2)


Windows 10 Wake On Lan fixes

helpful links from #29985:
http://www.groovypost.com/howto/enable-wake-on-lan-windows-10/
https://www.reddit.com/r/Windows10/comments/3f73sz/psaif_windows_10_killed_your_wol_functionality_or/


Worker Evaluation Questionnaire

This is deprecated as of 7/5/22 (see https://redmine.palantetech.coop/issues/66220). Use Worker_Evaluation_Process instead!

This worksheet is designed to be used by cooperative members to help inform a staff member's annual evaluation. Please be as specific as possible in your responses.

Name of staff member being evaluated:
Date: