---
title: "ihsMW User Guide"
subtitle: "Cleaning and harmonising Malawi Integrated Household Survey data in R"
author: "Vitumbiko Kayuni"
date: "`r format(Sys.Date(), '%d %B %Y')`"
output:
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 3
vignette: >
  %\VignetteIndexEntry{ihsMW User Guide}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.align = "center"
)
library(ihsMW)
```

# About this guide

This is the complete manual for **ihsMW**, an R package for cleaning,
harmonising and analysing the Malawi Integrated Household Survey (IHS).

It is written to be read start to finish by someone who has never used the
package, and then kept nearby as a reference. Every code block that does not
require microdata runs as written — the guide is generated by executing its own
examples, so what you see is what the package actually produces.

**Three ways to read it.** Online at
<https://vituk123.github.io/ihsMW/articles/ihsMW-user-guide.html>, as a
[PDF](https://vituk123.github.io/ihsMW/ihsMW-user-guide.pdf) for printing and
offline use, or as
[R Markdown source](https://vituk123.github.io/ihsMW/ihsMW-user-guide.Rmd) if
you want to re-run and adapt every example yourself.

**What this guide assumes.** Basic R: you can install a package, read a data
file, and subset a data frame. It does not assume you know anything about the
IHS, about survey weights, or about `dplyr`.

---

# 1. Installation

## From CRAN

```{r eval=FALSE}
install.packages("ihsMW")
```

## Development version

```{r eval=FALSE}
# install.packages("pak")
pak::pak("vituk123/ihsMW")

# or
# install.packages("remotes")
remotes::install_github("vituk123/ihsMW")
```

## Confirm the installation

```{r}
library(ihsMW)
packageVersion("ihsMW")
```

`ihsMW` requires **R 4.1.0 or later**, because it uses the native pipe
(`|>`) internally.

---

# 2. Package dependencies

`ihsMW` is deliberately light. Everything needed for the core cleaning and
harmonisation workflow is a hard dependency; anything optional is suggested,
so a minimal install stays small.

## Required (installed automatically)

| Package | Minimum | Used for |
|---|---|---|
| `dplyr` | 1.1.0 | joins, grouping, aggregation |
| `readr` | 2.1.0 | reading the bundled crosswalk and reference CSVs |
| `rlang` | 1.1.0 | argument matching and tidy evaluation |
| `cli` | 3.6.0 | the informative console messages and errors |

## Suggested (install if you need the feature)

| Package | Needed for |
|---|---|
| `haven` | reading Stata `.dta` microdata — **you will almost certainly want this** |
| `survey` | `ihs_svydesign()` and design-based standard errors |
| `srvyr` | `dplyr`-style syntax over survey designs |
| `testthat`, `withr`, `jsonlite` | running the package test suite |
| `knitr`, `rmarkdown`, `pkgdown` | rebuilding this guide and the website |

Install the ones most people want in one go:

```{r eval=FALSE}
install.packages(c("haven", "survey", "srvyr"))
```

## No internet required

Once installed, `ihsMW` never makes a network request. The crosswalk,
conversion factors and CPI series are all bundled inside the package. This is
deliberate: survey work often happens on airgapped or restricted machines, and
an analysis that silently depends on a live API is not reproducible.

---

# 3. Package overview

## The problem

The IHS is Malawi's flagship household survey, run roughly every five years by
the National Statistical Office with World Bank LSMS support. It is the
evidence base for poverty measurement, food security monitoring and
agricultural policy in Malawi.

Working with it raw is painful in specific, repetitive ways:

1. **Variable names drift between rounds.** The same concept is `hhweight` in
   IHS2 and IHS3 but `hh_wgt` from IHS4 onward. The design stratum is
   `stratum` through IHS5 and `strata` in IHS6. Multiply this across thousands
   of variables.
2. **Missing values are coded as numbers.** `-99` means "refused", `-98` means
   "don't know". Take a mean without recoding and your average household is
   younger than zero.
3. **Harvest quantities are in whatever unit the household used.** Pails,
   ox-carts, heaps, 50 kg bags — all in the same column. Summing it is
   meaningless without crop-, unit- and region-specific conversion factors.
4. **Money is nominal.** Malawi's price level rose roughly 24-fold between 2004
   and 2025. Comparing kwacha across rounds without deflating produces
   nonsense.
5. **The sample is not simple random.** It is a stratified two-stage cluster
   design. Ignore the weights and your estimates are biased; ignore the
   clustering and your standard errors are far too small.

Every research team solves these five problems again, in their own scripts,
with their own undocumented judgement calls. `ihsMW` solves them once.

## What the package does

Thirteen exported functions covering the pipeline from raw file to
publication table:

```{r echo=FALSE}
knitr::kable(data.frame(
  Stage = c("Discover", "Discover", "Discover", "Discover",
            "Clean", "Clean", "Clean",
            "Transform", "Transform", "Transform",
            "Analyse", "Analyse", "Analyse"),
  Function = c("ihs_search()", "ihs_crosswalk_check()", "ihs_panel_ids()",
               "ihs_harmonise()",
               "ihs_clean()", "ihs_standardize_missing()", "ihs_winsorize()",
               "ihs_convert_units()", "ihs_aggregate()", "ihs_merge()",
               "ihs_deflate()", "ihs_svydesign()", "ihs_report()"),
  Purpose = c(
    "Find a variable by keyword across every round",
    "See how much of the crosswalk is comparable across rounds",
    "Look up the ID, weight and stratum columns for a round",
    "Rename raw columns to consistent harmonised names",
    "Recode missing codes and winsorize, with an audit trail",
    "Turn -99/-98/999 into NA",
    "Cap outliers at percentiles, optionally within strata",
    "Convert harvest units to kilograms with NSO factors",
    "Roll individual or plot records up to the household",
    "Join modules on auto-detected ID columns",
    "Convert nominal kwacha to real, constant-price kwacha",
    "Build a survey design object for correct standard errors",
    "Produce a weighted summary statistics table"
  ),
  check.names = FALSE
))
```

## What the package does *not* do

Being explicit about the boundaries saves you time:

- **It does not distribute microdata.** World Bank licensing forbids it. You
  download the files yourself (see §5).
- **It does not construct a poverty line or a consumption aggregate.** Those
  are published with each round; the package helps you clean and compare them,
  not rebuild them.
- **It does not adjust for spatial price differences.** `ihs_deflate()` applies
  a national CPI over time. If you need a Lilongwe-vs-Mzuzu price adjustment,
  use the spatial price index published with the round's consumption
  aggregate.
- **It does not link households into a panel across rounds.** The IHS panel
  component has its own ID structure and attrition rules;
  `ihs_panel_ids()` tells you which columns to use, but the linking logic is
  yours.
- **It does not make judgement calls for you.** When a variable cannot be
  mapped or a unit cannot be converted, you get `NA` and a message — never a
  quietly invented number.

## Supported rounds

```{r}
ihs_panel_ids("all")
```

Note the two renames that catch people out: the weight column changes at IHS4,
and the stratum column changes at IHS6. `ihs_svydesign()` handles both
automatically.

IHS1 (1997/98) is not supported — it is not available in a form the crosswalk
can address.

---

# 4. Basic workflow

Almost every analysis follows the same seven steps.

```
  +------------------------+
  | 1. Read raw file       |   haven::read_dta() / readr::read_csv()
  +-----------+------------+
              v
  +------------------------+
  | 2. Harmonise names     |   ihs_harmonise(round = "IHS6")
  +-----------+------------+
              v
  +------------------------+
  | 3. Merge modules       |   ihs_merge()
  +-----------+------------+
              v
  +------------------------+
  | 4. Clean               |   ihs_clean()
  +-----------+------------+
              v
  +------------------------+
  | 5. Convert & deflate   |   ihs_convert_units() / ihs_deflate()
  +-----------+------------+
              v
  +------------------------+
  | 6. Survey design       |   ihs_svydesign()
  +-----------+------------+
              v
  +------------------------+
  | 7. Report              |   ihs_report() / survey::svymean()
  +------------------------+
```

The order matters in two places:

- **Harmonise before merging.** `ihs_merge()` detects join keys by name, and
  those names only line up after harmonisation.
- **Build the design before subsetting.** Filtering rows out of a data frame
  destroys the stratum and cluster structure. Create the design on the full
  data, then use `survey::subset()` on the design object.

---

# 5. Getting the data

The microdata are free but require registration.

1. Go to the [World Bank Microdata Library](https://microdata.worldbank.org).
2. Search for the round you want:

   | Round | Years | Catalogue title |
   |---|---|---|
   | IHS2 | 2004/05 | Malawi Second Integrated Household Survey |
   | IHS3 | 2010/11 | Malawi Third Integrated Household Survey |
   | IHS4 | 2016/17 | Malawi Fourth Integrated Household Survey |
   | IHS5 | 2019/20 | Malawi Fifth Integrated Household Survey |
   | IHS6 | 2024/25 | Malawi Sixth Integrated Household Survey |

3. Request access, stating your research purpose. Approval is usually quick.
4. Download the **Stata (`.dta`)** distribution if you have a choice — see the
   warning below.
5. Unzip into a folder per round.

## Prefer the Stata distribution

Some IHS CSV exports substitute *value labels* for numeric codes. In the IHS5
CSV release, for instance, `crop_code` contains `"MAIZE LOCAL"` rather than
`1`, and the harvest unit contains `"50 KG BAG"` rather than `2`. The
conversion factor table is keyed on the numeric codes, so unit conversion
cannot work on those files.

`ihs_convert_units()` detects this and stops with an explanatory error rather
than handing you a column of `NA`. The fix is to read the `.dta` version:

```{r eval=FALSE}
library(haven)
ag <- haven::read_dta("IHS5/ag_mod_i.dta")   # numeric codes, with labels attached
```

`haven` also preserves variable labels, which `ihs_harmonise()` carries through
the rename.

---

# 6. Input data requirements

## Format

Any `data.frame` works, including a `tibble` or the labelled data frame that
`haven::read_dta()` returns. `ihsMW` returns the same class it was given.

## What each function needs

| Function | Requires |
|---|---|
| `ihs_harmonise()` | Any data frame; column names matching the round's crosswalk entries |
| `ihs_merge()` | Two or more data frames sharing at least one ID column |
| `ihs_standardize_missing()` | Any data frame; only numeric columns are touched |
| `ihs_winsorize()` | Numeric target columns; the `by` column must exist |
| `ihs_convert_units()` | Numeric `crop`, `unit` and quantity columns |
| `ihs_aggregate()` | A grouping column present in the data |
| `ihs_deflate()` | Numeric value columns plus a `round` argument or an `ihs_round` column |
| `ihs_svydesign()` | A weight column; strata and PSU columns if you want them used |
| `ihs_report()` | Numeric variables; grouping and weight columns if used |

## Column naming

After `ihs_harmonise()`, columns carry the crosswalk's `harmonised_name`.
Downstream functions detect standard columns case-insensitively, so `HHID` and
`hhid` are equivalent. The names each function looks for:

- **Weights:** `hh_wgt`, `hhweight`, `weight`, `panelweight`, `hh_wgt_adj`, `hhwght`
- **Strata:** `stratum`, `strata`, `strataid`, `strat`
- **PSU / cluster:** `ea_id`, `psu`, `cluster`, `clusterid`
- **Join keys:** `case_id`, `hhid`, `hh_id`, `HHID`, `ea_id`, `PID`

If your columns are named differently, pass them explicitly — every
auto-detecting function accepts an override.

---

# 7. Output structure

## New columns, never overwritten values

`ihsMW` never modifies a value in place. Transformations add a suffixed column
and leave the original intact, so you can always audit or revert:

| Function | Suffix | Example |
|---|---|---|
| `ihs_winsorize()` | `_w` | `food_exp` → `food_exp_w` |
| `ihs_convert_units()` | `_kg` | `quantity` → `quantity_kg` |
| `ihs_deflate()` | `_real` | `food_exp` → `food_exp_real` |

Suffixes compose, which is how you tell at a glance what has been done:
`rexp_cat011_w_real` is a winsorized, then deflated, food consumption
variable.

`ihs_harmonise()` is the exception — renaming columns is its entire purpose.
It also adds an `ihs_round` column, which `ihs_deflate()` later reads.

## Audit attributes

Several functions attach a machine-readable record of what they did:

```{r}
survey_data <- data.frame(
  region   = rep(c("North", "South"), each = 50),
  food_exp = c(rnorm(50, 100, 10), rnorm(50, 500, 50))
)
survey_data$food_exp[1]  <- -99      # a refusal code
survey_data$food_exp[60] <- 99999    # an implausible outlier

cleaned <- ihs_clean(survey_data,
                     winsorize_vars = "food_exp",
                     winsorize_by   = "region")

str(attr(cleaned, "ihs_audit"))
```

The audit tells you 1 value was recoded to `NA` and how many observations were
capped at each bound. Put this in your appendix and reviewers stop asking how
you handled outliers.

Attributes available:

| Attribute | Set by | Contains |
|---|---|---|
| `ihs_audit` | `ihs_clean()` | Row/column counts before and after, plus both logs below |
| `ihs_missing_conversions` | `ihs_standardize_missing()` | Count of values recoded per column |
| `ihs_winsorized_vars` | `ihs_winsorize()` | Count capped at lower and upper bound per variable |
| `ihs_merge_log` | `ihs_merge()` | Row counts at each merge step |

---

# 8. Function reference

Every exported function, in workflow order.

## 8.1 `ihs_search()` — find a variable

```r
ihs_search(keyword, round = NULL, fields = c("name", "label", "module"))
```

Searches the bundled crosswalk. Use it before writing any code, to find out
what your variable is called in each round.

```{r}
res <- ihs_search("household size")
res[, c("harmonised_name", "label", "ihs2_name", "ihs5_name", "ihs6_name", "n_rounds")]
```

Restrict to a round to see only variables that exist in it:

```{r}
nrow(ihs_search("livestock", round = "IHS6"))
```

Search a single field to cut noise:

```{r}
nrow(ihs_search("expenditure", fields = "label"))
```

**Returns** a tibble with one row per matching variable, one `ihs*_name`
column per round, `n_rounds` (how many rounds it appears in) and
`needs_review`.

**Watch out:** `keyword` is a regular expression, so `.` and `(` are special.
Search for `"exp"`, not `"exp("`.

---

## 8.2 `ihs_crosswalk_check()` — audit cross-round comparability

```r
ihs_crosswalk_check(verbose = TRUE)
```

Prints a coverage report and returns the full crosswalk. Run it before
committing to a pooled multi-round analysis.

```{r}
cw <- ihs_crosswalk_check(verbose = FALSE)

nrow(cw)                    # total harmonised variables
table(cw$n_rounds_avail)    # how many rounds each appears in
```

Read that table carefully. Coverage is **not** uniform — only a small core of
variables appears in all five rounds, and most appear in one or two. Any claim
of a "consistent 20-year series" needs to survive this table first.

```{r}
# Variables available in every round
head(cw$harmonised_name[cw$n_rounds_avail == 5], 15)
```

### Why full-series coverage is so thin

Two things drive it, and it is worth knowing which is which:

1. **IHS2 is a much smaller instrument.** It has roughly 1,276 variables
   against IHS6's 3,214. A large share of what later rounds ask simply was not
   asked in 2004.
2. **The crosswalk is built mostly by matching variable names.** About 98% of
   its rows link rounds that happen to use the *same* name. IHS2 used an
   entirely different naming scheme (`ca15`, `cd37`, `sec_a`), so only 85 of
   its 1,272 mapped variables currently link to a later round.

Point 2 means the all-five-rounds count is a **lower bound**, not a ceiling.
Label-based matching would recover more IHS2 links, and contributions on that
front are welcome — see §15. Coverage from IHS3 onwards is much better:

```{r}
c(
  `IHS3+IHS4+IHS5`      = sum(!is.na(cw$ihs3_name) & !is.na(cw$ihs4_name) &
                              !is.na(cw$ihs5_name)),
  `IHS4+IHS5+IHS6`      = sum(!is.na(cw$ihs4_name) & !is.na(cw$ihs5_name) &
                              !is.na(cw$ihs6_name)),
  `all five rounds`     = sum(cw$n_rounds_avail == 5)
)
```

If your question can be answered from IHS3 onwards, you have far more to work
with than the all-five figure suggests.

### `needs_review` and `ihs6_expansion_of`

Two columns describe how much to trust an IHS6 mapping:

- `ihs6_expansion_of` names the parent question when IHS6 split a "select all
  that apply" item into one binary column per option. IHS5's single `ag_e07`
  became `ag_e07__1`, `ag_e07__4` and so on. These are **not** new concepts,
  and there are 342 of them.
- `needs_review` marks a variable that is new in IHS6 and whose concept has not
  been cross-checked against an earlier instrument. Confirm what it measures
  before pooling it across rounds.

```{r}
# Multi-select expansions, with their parent question
exp <- cw[!is.na(cw$ihs6_expansion_of),
          c("harmonised_name", "ihs6_expansion_of")]
head(exp, 5)
nrow(exp)
```

---

## 8.3 `ihs_panel_ids()` — look up design columns

```r
ihs_panel_ids(round = "IHS5")
```

```{r}
ihs_panel_ids("IHS6")
```

Use `"all"` for the cross-round comparison shown in §3.

---

## 8.4 `ihs_harmonise()` — rename to standard names

```r
ihs_harmonise(data, round = "IHS5", extra = FALSE)
```

The foundation of everything else. Renames raw columns to harmonised names
using the crosswalk, preserving Stata variable labels, and adds an
`ihs_round` column.

```{r}
raw <- data.frame(
  case_id = c("a", "b"),
  hhsize  = c(4, 6),
  junk    = c(1, 2)
)

ihs_harmonise(raw, round = "IHS6")
```

By default, columns absent from the crosswalk are dropped. Keep everything
with `extra = TRUE`:

```{r}
ihs_harmonise(raw, round = "IHS6", extra = TRUE)
```

**Watch out:**

- Matching is case-insensitive, and each raw column is claimed by at most one
  crosswalk entry.
- A warning that *no* columns mapped almost always means the wrong `round`.
- `extra = FALSE` is destructive. If you need a variable that is not in the
  crosswalk, use `extra = TRUE` and open an issue so it can be added.

---

## 8.5 `ihs_merge()` — join modules

```r
ihs_merge(..., by = NULL, type = "left")
```

```{r}
hh <- data.frame(case_id = c("A", "B", "C"), hhsize = c(4, 6, 3))
ag <- data.frame(case_id = c("A", "B", "D"), harvest_kg = c(120, 340, 90))

merged <- ihs_merge(hh, ag)
merged
```

`type` may be `"left"` (default), `"inner"` or `"full"`. Row counts at each
step are recorded:

```{r}
attr(merged, "ihs_merge_log")
```

**Watch out:** columns present in more than one input but not used as join
keys get dplyr's `.x`/`.y` suffixes. `ihs_merge()` warns when this happens,
because it silently breaks downstream auto-detection — once `hh_wgt` becomes
`hh_wgt.x`, `ihs_svydesign()` can no longer find the survey weight:

```{r}
a <- data.frame(case_id = c("A", "B"), region = 1:2, x = 1:2)
b <- data.frame(case_id = c("A", "B"), region = 1:2, y = 3:4)

names(suppressWarnings(ihs_merge(a, b)))
```

Fix it by dropping the duplicate before merging, or by adding the shared
column to `by`:

```{r}
names(ihs_merge(a, b, by = c("case_id", "region")))
```

---

## 8.6 `ihs_standardize_missing()` — recode missing codes

```r
ihs_standardize_missing(data)
```

Recodes `-99`, `-98`, `-97`, `999`, `998` and `997` to `NA` in numeric
columns.

```{r}
df <- data.frame(age = c(34, -99, 41, 998), village = c("A", "B", "C", "D"))
clean <- ihs_standardize_missing(df)
clean$age
attr(clean, "ihs_missing_conversions")
```

**Watch out:** `999` is a legitimate value for some variables — a plot area in
square metres, a price in kwacha. This is a blanket recode. Check your
variables first, and if in doubt pass only the affected columns.

---

## 8.7 `ihs_winsorize()` — cap outliers

```r
ihs_winsorize(data, vars, by = NULL, probs = c(0.01, 0.99))
```

Caps values below the lower and above the upper percentile, writing the result
to a new `_w` column.

```{r}
set.seed(42)
df <- data.frame(
  region = rep(c("North", "South"), each = 50),
  cons   = c(rnorm(50, 100, 10), rnorm(50, 500, 50))
)

global    <- ihs_winsorize(df, vars = "cons", probs = c(0.05, 0.95))
by_region <- ihs_winsorize(df, vars = "cons", by = "region",
                           probs = c(0.05, 0.95))

# The raw column is untouched either way
identical(global$cons, df$cons)
```

**Why stratify.** A single national 99th percentile treats the richest rural
households as outliers, because the urban distribution sits far above them.
Compare the maximum retained in the North under each approach:

```{r}
c(global    = max(global$cons_w[global$region == "North"]),
  stratified = max(by_region$cons_w[by_region$region == "North"]))
```

Stratifying preserves the shape of both distributions. This matters
enormously for inequality and poverty measures.

---

## 8.8 `ihs_clean()` — the wrapper

```r
ihs_clean(data, winsorize_vars = NULL, winsorize_by = NULL,
          probs = c(0.01, 0.99))
```

Runs `ihs_standardize_missing()` and then `ihs_winsorize()`, collecting both
audit logs into a single `ihs_audit` attribute. Use it unless you need finer
control.

```{r}
result <- ihs_clean(df, winsorize_vars = "cons", winsorize_by = "region")
names(attr(result, "ihs_audit"))
```

**Watch out:** the argument is `winsorize_vars`, not `winsorize_cols`, and the
grouping argument is `winsorize_by`, not `strata_col`. There is no
`missing_cols` argument — missing-code recoding always applies to every
numeric column.

---

## 8.9 `ihs_convert_units()` — harvest units to kilograms

```r
ihs_convert_units(data, qty_col, unit_col, crop_col, unmapped = "warn")
```

Converts reported harvest quantities to kilograms using 1,608 official NSO
conversion factors, keyed on crop, unit, region and shelling condition.

```{r}
harvest <- data.frame(
  crop_code = c(1, 1, 1),
  unit_code = c(1, 2, 3),
  quantity  = c(100, 2, 5),
  region    = c(1, 2, 2)
)

ihs_convert_units(harvest, qty_col = "quantity",
                  unit_col = "unit_code", crop_col = "crop_code")
```

Two pails of maize is 100 kg; five of the next unit up is 450 kg. Summing the
raw `quantity` column would have given 107 of nothing in particular.

**Matching rules,** applied in order:

1. exact match on crop, unit, region and condition;
2. same crop, unit and region, falling back through the condition preference
   order shelled (1) → not applicable (3) → unshelled (2);
3. the same two steps against the Central region, used as the national
   fallback when a crop-unit pair has no factor for the row's own region.

A `region` column is detected automatically. Without one, every row is priced
at Central-region factors and the function says so — applying Central factors
to Northern or Southern harvests biases every converted quantity.

There is one exception to the crop-specific rule. Unit code 1 is KILOGRAM, and
a kilogram is a kilogram whatever the crop, so quantities already reported in
kilograms convert even for crops the factor table does not list — including
those introduced in IHS6:

```{r}
kg <- data.frame(crop_code = c(1, 48), unit_code = c(1, 1), quantity = c(7, 40))
ihs_convert_units(kg, "quantity", "unit_code", "crop_code")
```

The package verifies this against the bundled table rather than assuming it: if
a future NSO release gave the kilogram unit a crop-specific factor, the rule
switches itself off.

**Unmappable combinations return `NA`,** never a guess, and the function names
them:

```{r}
odd <- data.frame(crop_code = 999, unit_code = 999, quantity = 10)
suppressWarnings(
  ihs_convert_units(odd, "quantity", "unit_code", "crop_code")
)
```

Set `unmapped = "error"` to make failures fatal in a production pipeline, or
`"ignore"` to silence the warning once you have understood it.

### Known gaps in the factor table

On IHS6 crop sales, about 10% of otherwise-convertible rows still return `NA`.
The causes, in order of size:

- **Units with no published factor at all** — code 6 (No. 10 plate), 7 (No. 12
  plate) and 10 (bale) are used in IHS6 but absent from the NSO table.
- **Crops introduced in IHS6** — codes 42, 48, 49 and 50 have no factors for
  non-kilogram units.
- **Unit 13, "OTHER (SPECIFY)"** — unmappable by design. The household named
  its own unit, so no general factor can exist. Roughly 113 rows in
  `ag_mod_i`.

Do not paper over these. `NA` is the correct answer, and a total computed over
the rows that *did* convert is honest as long as you report the coverage. If
you have an official NSO factor for any of these, please
[contribute it](https://github.com/vituk123/ihsMW/issues/new/choose) — the
repository has an issue template for exactly this.

---

## 8.10 `ihs_aggregate()` — roll up to the household

```r
ihs_aggregate(data, group_col = "case_id")
```

```{r}
members <- data.frame(
  case_id   = c("A", "A", "B", "B", "B"),
  income    = c(1000, 500, 200, 300, 250),
  has_radio = c(0, 1, 0, 0, 0),
  district  = c("Lilongwe", "Lilongwe", "Blantyre", "Blantyre", "Blantyre")
)

ihs_aggregate(members, group_col = "case_id")
```

**Rules.** Continuous numeric columns are summed; 0/1 columns become a logical
OR; logical columns become `any()`; character and factor columns take their
single distinct value, or the distinct values joined with `" | "`; anything
else is dropped with a warning.

**Watch out:** summing is right for quantities and expenditures and wrong for
ages, prices and rates. Select your columns before calling, or aggregate those
yourself.

---

## 8.11 `ihs_deflate()` — nominal to real

```r
ihs_deflate(data, value_cols, round = NULL, base_year = 2019)
```

Converts nominal kwacha to constant-price kwacha so figures from different
rounds can be compared.

```{r}
pooled <- data.frame(
  food_exp  = c(1000, 1000, 1000, 1000, 1000),
  ihs_round = c("IHS2", "IHS3", "IHS4", "IHS5", "IHS6")
)

ihs_deflate(pooled, value_cols = "food_exp")
```

1,000 kwacha in 2004 bought what roughly 7,500 bought in 2019; 1,000 kwacha in
2024 bought about 409-worth. Comparing the nominal column across rounds would
have reversed the story entirely.

Because `ihs_harmonise()` adds `ihs_round`, pooled data deflates in one call
with no arguments. For a single-round frame, name the round:

```{r}
ihs_deflate(data.frame(v = 1000), value_cols = "v", round = "IHS6")
```

**The CPI series.** World Bank WDI `FP.CPI.TOTL` for Malawi, rebased to
2019 = 100, covering 2004–2025. Rounds map to their survey midpoint year:
IHS2 → 2004, IHS3 → 2010, IHS4 → 2016, IHS5 → 2019, IHS6 → 2024.

The table records where each figure came from and whether it is still liable
to revision:

```{r}
cpi <- read.csv(system.file("extdata", "mw_cpi_annual.csv", package = "ihsMW"))
tail(cpi[, c("year", "cpi_index", "retrieved", "provisional")], 3)
```

The World Bank revises recent observations as national statistics are
finalised, so the two most recent years are marked `provisional`.
`ihs_deflate()` says so whenever a round you are deflating depends on one — as
IHS6 (2024) currently does. These are the best figures available; just be aware
they can move, and record which vintage you used. Refresh with
`data-raw/02_build_cpi.R`.

**Watch out:** this is a **national** deflator over time. It does not adjust
for price differences between districts or between urban and rural areas. If
your analysis is sensitive to those, each round's consumption aggregate
publishes its own spatial index — IHS6 ships `spatial_indexL` (regional price
level) and `price_indexL` (combined spatial-temporal) — and you should divide
by that rather than relying on this function alone.

---

## 8.12 `ihs_svydesign()` — build the survey design

```r
ihs_svydesign(data, weight_col = NULL, strata_col = NULL, psu_col = NULL)
```

Wraps `survey::svydesign()`, detecting the weight, stratum and PSU columns.

```{r eval=requireNamespace("survey", quietly = TRUE)}
hh <- data.frame(
  case_id  = 1:20,
  ea_id    = rep(1:5, each = 4),
  strata   = rep(c("urban", "rural"), each = 10),
  hh_wgt   = runif(20, 0.5, 3),
  food_exp = rnorm(20, 5000, 1000)
)

dsgn <- ihs_svydesign(hh)
survey::svymean(~food_exp, dsgn, na.rm = TRUE)
```

The same call works on every round: case-insensitive matching handles
IHS2/IHS3 `hhweight`, IHS4+ `hh_wgt`, IHS2–IHS5 `stratum` and IHS6 `strata`.

If strata or PSU are missing the function degrades gracefully — to a clustered
design, then to a simple weighted one — and tells you which it built. Read
that message: a simple weighted design gives correct point estimates but
standard errors that are too small.

---

## 8.13 `ihs_report()` — summary table

```r
ihs_report(data, vars = NULL, by = NULL, weights = NULL)
```

```{r}
hh <- data.frame(
  hhsize   = c(4, 6, 3, 5, 7, 2),
  food_exp = c(1200, 3400, 900, 2100, 4500, 700),
  region   = c("North", "North", "Central", "Central", "South", "South"),
  hh_wgt   = c(1.2, 0.8, 1.5, 1.1, 0.9, 1.3)
)

ihs_report(hh, vars = c("hhsize", "food_exp"))
```

Group and weight it:

```{r}
ihs_report(hh, vars = "food_exp", by = "region", weights = "hh_wgt")
```

With `vars = NULL` every numeric column is summarised, with weight-like
columns excluded automatically.

**Watch out:** only the mean and SD are weighted. `n`, `median`, `min`, `max`
and `pct_missing` describe the sample, not the population. For weighted
quantiles or design-correct standard errors, use `ihs_svydesign()` and the
`survey` package.

---

# 9. Complete worked workflows

These use real file names. Substitute your own download paths.

## Workflow A — one round, weighted poverty statistics

```{r eval=FALSE}
library(ihsMW)
library(haven)
library(survey)

# 1. Read
demog <- haven::read_dta("IHS6/hh_mod_a_filt.dta")
cons  <- haven::read_dta("IHS6/ihs6_consumptionaggregates.dta")

# 2. Harmonise (adds `ihs_round`)
demog_h <- ihs_harmonise(demog, round = "IHS6")
cons_h  <- ihs_harmonise(cons,  round = "IHS6")

# 3. Merge, dropping columns that appear in both to avoid .x/.y suffixes
dupes  <- c("region", "district", "hhsize", "hh_wgt", "ihs_round")
merged <- ihs_merge(demog_h, cons_h[, setdiff(names(cons_h), dupes)])

# 4. Clean: recode missing codes, winsorize within region
clean <- ihs_clean(merged,
                   winsorize_vars = "rexp_cat011",
                   winsorize_by   = "region")

# 5. Deflate to 2019 prices
real <- ihs_deflate(clean, value_cols = "rexp_cat011_w")

# 6. Survey design
dsgn <- ihs_svydesign(real)

# 7. Report
ihs_report(real,
           vars    = c("hhsize", "rexp_cat011_w_real"),
           by      = "region",
           weights = "hh_wgt")

# Design-correct national mean with a standard error
survey::svymean(~rexp_cat011_w_real, dsgn, na.rm = TRUE)

# Document what cleaning did, for your appendix
str(attr(clean, "ihs_audit"))
```

## Workflow B — pooling IHS4, IHS5 and IHS6

```{r eval=FALSE}
library(ihsMW)
library(haven)
library(dplyr)

read_round <- function(path, round) {
  haven::read_dta(path) |> ihs_harmonise(round = round)
}

ihs4 <- read_round("IHS4/ihs4_consumption_aggregate.dta", "IHS4")
ihs5 <- read_round("IHS5/ihs5_consumption_aggregate.dta", "IHS5")
ihs6 <- read_round("IHS6/ihs6_consumptionaggregates.dta", "IHS6")

# Check comparability BEFORE pooling
cw <- ihs_crosswalk_check(verbose = FALSE)
comparable <- cw$harmonised_name[
  !is.na(cw$ihs4_name) & !is.na(cw$ihs5_name) & !is.na(cw$ihs6_name)
]

keep <- intersect(comparable, Reduce(intersect, list(names(ihs4), names(ihs5), names(ihs6))))
keep <- union(keep, "ihs_round")

pooled <- bind_rows(ihs4[, keep], ihs5[, keep], ihs6[, keep])

# One call deflates every round correctly, using `ihs_round`
pooled <- ihs_deflate(pooled, value_cols = "rexp_cat011")

# Real food consumption by round, on a common 2019 basis
ihs_report(pooled, vars = "rexp_cat011_real", by = "ihs_round",
           weights = "hh_wgt")
```

The `comparable` step is the important one. Pooling variables that do not
actually exist in all three rounds produces a series with silent structural
breaks.

## Workflow C — agricultural production per household

```{r eval=FALSE}
library(ihsMW)
library(haven)

# Plot-level harvest, and the household roster for region
harvest <- haven::read_dta("IHS6/ag_mod_g.dta")
roster  <- haven::read_dta("IHS6/hh_mod_a_filt.dta")

# Attach region so region-specific conversion factors are used
harvest <- ihs_merge(harvest, roster[, c("case_id", "region")], by = "case_id")

# Convert mixed units to kilograms
harvest_kg <- ihs_convert_units(
  harvest,
  qty_col  = "ag_g13a",
  unit_col = "ag_g13b",
  crop_col = "crop_code",
  unmapped = "warn"
)

# Sum to household level
hh_production <- ihs_aggregate(
  harvest_kg[, c("case_id", "ag_g13a_kg")],
  group_col = "case_id"
)

head(hh_production)
```

Note the merge in step two: without a `region` column, every row is converted
at Central-region factors.

---

# 10. Common mistakes

**1. Using the wrong argument names.** The three that trip people up most:

```{r eval=FALSE}
# WRONG
ihs_clean(df, winsorize_cols = "x", strata_col = "region")
ihs_winsorize(df, value_col = "x", strata_col = "region")
ihs_aggregate(df, id_cols = "case_id", val_cols = "x")

# RIGHT
ihs_clean(df, winsorize_vars = "x", winsorize_by = "region")
ihs_winsorize(df, vars = "x", by = "region")
ihs_aggregate(df, group_col = "case_id")
```

**2. Merging before harmonising.** Join keys are detected by name, and the
names only line up after `ihs_harmonise()`.

**3. Analysing the raw column after winsorizing.** `ihs_winsorize()` writes to
`x_w` and leaves `x` alone. If your results look unchanged, you are probably
still using `x`.

**4. Subsetting before building the survey design.** This breaks the stratum
and cluster structure and gives wrong standard errors:

```{r eval=FALSE}
# WRONG
rural <- data[data$urban == 0, ]
dsgn  <- ihs_svydesign(rural)

# RIGHT
dsgn  <- ihs_svydesign(data)
rural <- subset(dsgn, urban == 0)
```

**5. Comparing nominal kwacha across rounds.** Always `ihs_deflate()` first.
Malawi's price level rose roughly 24-fold between 2004 and 2025; nominal
comparisons across rounds are not merely imprecise, they are usually backwards.

**6. Assuming every variable exists in every round.** Check
`ihs_crosswalk_check()` first. Only a small core spans all five rounds.

**7. Ignoring the `.x`/`.y` warning from `ihs_merge()`.** It is telling you
that a column you rely on — often the survey weight — has been renamed.

**8. Summing quantities before converting units.** A column mixing pails,
ox-carts and 50 kg bags cannot be summed. Convert first.

**9. Using CSV files for unit conversion.** Some IHS CSV exports contain value
labels instead of numeric codes. Use the `.dta` distribution.

**10. Treating `-99` as a real number.** Run `ihs_standardize_missing()` — or
`ihs_clean()`, which includes it — before any arithmetic.

---

# 11. Troubleshooting

| Message | Cause | Fix |
|---|---|---|
| `Invalid round(s) specified: "IHS7"` | Unsupported round | Use IHS2–IHS6. IHS1 is not available. |
| `No columns were mapped to harmonised names` | Wrong `round`, or a file whose variables are not in the crosswalk | Check the `round` argument first; then `ihs_search()` for one of your column names |
| `No common ID columns detected across all data.frames` | Inputs were not harmonised, or genuinely share no key | Harmonise first, or pass `by` explicitly |
| `N columns appeared in more than one input and were suffixed` | Shared non-key columns got `.x`/`.y` | Drop the duplicates, or add them to `by` |
| `Merge step 1 expanded rows from X to Y` | Many-to-many join | Aggregate one side to household level first, or add a finer key |
| `Column 'crop_code' holds value labels, not numeric codes` | CSV export substituted labels for codes | Read the `.dta` version with `haven::read_dta()` |
| `Failed to map N crop-unit combinations` | No published NSO factor for that pair | Expected for new IHS6 codes; those rows get `NA`. Report the combination as an issue |
| `Could not detect a survey weight column` | Weight renamed by a merge, or absent | Check for `hh_wgt.x`; pass `weight_col` explicitly |
| `Grouping variable 'region' not found in data` | Column dropped by `ihs_harmonise(extra = FALSE)` or suffixed by a merge | Check `names(data)` |
| `Base year 2019 not found in CPI table` | `base_year` outside 2004–2025 | Choose a year within range |
| `Cannot determine IHS round` | No `round` argument and no `ihs_round` column | Pass `round`, or harmonise first |

## Diagnosing anything else

```{r eval=FALSE}
# What do I actually have?
names(data)
str(data[, 1:10])

# Did harmonisation do anything?
sum(names(data) %in% ihs_crosswalk_check(verbose = FALSE)$harmonised_name)

# Where did my rows go?
attr(merged, "ihs_merge_log")

# What did cleaning change?
str(attr(cleaned, "ihs_audit"))
```

---

# 12. Frequently asked questions

**Does the package include the survey data?**
No. World Bank licensing prohibits redistribution. Download it yourself (§5).
Everything else — crosswalk, conversion factors, CPI — is bundled.

**Does it need an internet connection?**
No. `ihsMW` never makes a network request after installation.

**Which rounds are supported?**
IHS2 (2004/05), IHS3 (2010/11), IHS4 (2016/17), IHS5 (2019/20) and IHS6
(2024/25). IHS1 (1997/98) is not available in a usable form.

**Can I compare a variable across all five rounds?**
Only if it exists in all five, and only 49 do. Check with
`ihs_crosswalk_check()` before designing your analysis. Coverage from IHS3
onwards is far better — see §8.2 for why, and for the numbers.

**Why did `ihs_convert_units()` return `NA` for a quantity in kilograms?**
It should not, and no longer does: kilogram quantities convert even for crops
the factor table does not list. If you still see this, you are on a version
before 1.1.0.

**What changed in IHS6?**
Three things matter. The design stratum was renamed `stratum` → `strata`
(handled automatically). `adulteq` (adult equivalents) is absent from the
consumption aggregate, so per-adult-equivalent measures must be constructed
yourself. And consumption aggregates are published in July 2024 prices — use
`ihs_deflate()` to put every round on a common basis.

**A variable I need is missing from the crosswalk.**
Use `ihs_harmonise(extra = TRUE)` to keep it, then open a
[crosswalk mapping issue](https://github.com/vituk123/ihsMW/issues/new/choose)
so it is available to everyone next release.

**Why did `ihs_convert_units()` return all `NA`?**
Either your crop/unit columns hold value labels rather than numeric codes (the
function now errors rather than returning `NA` for this), or the crop-unit
combinations have no published factor. The warning names them.

**Should I winsorize at 1% or 5%?**
There is no universal answer. `c(0.01, 0.99)` is the common default for
consumption. Report whichever you used, and check that your conclusions
survive the alternative.

**Should I winsorize before or after deflating?**
Before. Winsorizing bounds the nominal distribution within its own round;
deflating afterwards is a monotone rescaling that preserves those bounds.

**Can I use `ihsMW` with `srvyr`?**
Yes. `ihs_svydesign()` returns a standard `survey.design2`, so
`srvyr::as_survey(dsgn)` works directly.

**Is the CPI series the official NSO one?**
It is the World Bank WDI series `FP.CPI.TOTL` for Malawi, rebased to
2019 = 100. WDI is compiled from NSO returns but the two can differ slightly
in revision timing. If your work requires the NSO series exactly, substitute
it — the file is a plain CSV at
`system.file("extdata", "mw_cpi_annual.csv", package = "ihsMW")`.

**How do I cite this?**
See §14.

---

# 12a. Known limitations

Stated plainly, so you can judge whether they affect your work. None of these
are hidden by the package — each one surfaces as a message, an `NA`, or a
column you can inspect.

| Limitation | What it means for you | Status |
|---|---|---|
| **Thin full-series coverage.** Only 49 variables span all five rounds | IHS2 is a genuinely smaller instrument, but the crosswalk also under-links it because it matches mostly on variable *names* and IHS2 used a different scheme | Improvable — label-based matching would recover more. Contributions welcome |
| **~270 IHS6 variables flagged `needs_review`** | New in IHS6 and not cross-checked against an earlier instrument. Confirm the concept before pooling across rounds | Inspect via the `needs_review` column |
| **~10% of IHS6 crop sales rows do not convert** | Units 6, 7 and 10 and crops 42, 48–50 have no published NSO factor; unit 13 is "OTHER (SPECIFY)" and never will | Partly fixable with official NSO data |
| **No spatial price adjustment** | `ihs_deflate()` is a national deflator over time only | Use the round's own `spatial_indexL` if you need it |
| **The two most recent CPI years are provisional** | Real values for IHS6 can shift when the World Bank revises 2024 | `ihs_deflate()` tells you; refresh with `data-raw/02_build_cpi.R` |
| **No cross-round household panel linking** | `ihs_panel_ids()` gives you the ID columns, but the linking and attrition logic is yours | Out of scope by design |

The general principle: where `ihsMW` cannot give you a defensible answer it
gives you `NA` and says why. It never invents a number to fill a gap.

---

# 13. Best practices

**Script everything; click nothing.** The whole point of `ihsMW` is that your
cleaning decisions live in version-controlled code rather than in a colleague's
memory.

**Keep the audit trail.** Save `attr(x, "ihs_audit")` alongside your results.
It answers the outlier and missing-data questions reviewers always ask.

**Check comparability before pooling.** Run `ihs_crosswalk_check()` and filter
to variables that genuinely exist in every round you use.

**Deflate before comparing money across rounds.** Every time.

**Winsorize within strata.** A national threshold systematically over-trims
the top of poorer strata. Pass `by`.

**Build the design once, on the full data.** Subset the design object, never
the data frame.

**Read the console messages.** `ihsMW` is deliberately chatty about what it
auto-detected. The messages tell you which weight column it found and which
join keys it used — the two things most likely to be silently wrong.

**Report your parameters.** Winsorization percentiles, base year, which rounds
and which variables. A reader should be able to reproduce your table from your
methods section.

**Pin your versions.** Record `packageVersion("ihsMW")` in your output, or use
`renv`. The crosswalk grows between releases.

**Prefer `.dta` over `.csv`.** Numeric codes and preserved variable labels.

---

# 14. Citation

To cite the package:

```{r eval=FALSE}
citation("ihsMW")
```

```bibtex
@Manual{ihsMW,
  title  = {ihsMW: Clean and Harmonise Malawi Integrated Household Survey Data},
  author = {Vitumbiko Kayuni},
  year   = {2026},
  note   = {R package version 1.1.1},
  url    = {https://github.com/vituk123/ihsMW},
}
```

**Also cite the data.** `ihsMW` is a tool, not a source. Any publication using
IHS microdata must cite the National Statistical Office of Malawi and the
World Bank LSMS programme. Each round's Basic Information Document gives the
exact required form — use it, because the requirement is a condition of your
data access agreement.

**Cite the reference data if you rely on it.** The crop conversion factors are
NSO's; the CPI series is World Bank WDI `FP.CPI.TOTL`.

---

# 15. Getting help and contributing

- **Documentation:** <https://vituk123.github.io/ihsMW/>
- **Bug reports:** <https://github.com/vituk123/ihsMW/issues>
- **Crosswalk mappings and conversion factors:** the repository has dedicated
  issue templates. A mapping you add helps everyone working on the IHS.

When reporting a bug, include the output of `sessionInfo()`, the version from
`packageVersion("ihsMW")`, and a minimal reproducible example using synthetic
data rather than microdata you are not licensed to share.

---

```{r echo=FALSE}
cat("Generated with ihsMW", as.character(packageVersion("ihsMW")),
    "on", format(Sys.Date(), "%d %B %Y"), "\n")
```