varwg.core package

Submodules

varwg.core.base module

class varwg.core.base.Base(var_names, met_file=None, sum_interval=24, max_nans=12, plot=False, separator='\t', refit=None, detrend_vars=None, verbose=False, data_dir=None, cache_dir=None, dump_data=True, non_rain=None, rain_method=None, neg_rain_doy_width=30, neg_rain_fft_order=2, neg_kwds=None, infill=False, fit_kwds=None, **met_kwds)

Bases: object

Handle everything except plotting.

Attributes:

K

The number of variables.

T_data

Length of the input data time series (‘hourly’).

T_summed

Length of aggregated input data (‘daily’).

data_doys

Days of year of aggregated input data.

data_doys_raw

Days of year of raw (hourly) input data.

dis_doys

Days of year of disaggregated output data.

output_resolution

In hours.

primary_var_ii

The row index of the primary variable.

sim_doys

Days of year of aggregated output data.

start_hour_of_src

sum_interval_dict

Maps var_name to sum_interval.

Methods

fitted_medians(var_name[, doys])

Medians of the fitted seasonal distribution.

dist_sol_hourly

property K

The number of variables.

property T_data

Length of the input data time series (‘hourly’).

property T_summed

Length of aggregated input data (‘daily’).

property data_doys

Days of year of aggregated input data.

property data_doys_raw

Days of year of raw (hourly) input data.

property dis_doys

Days of year of disaggregated output data.

dist_sol_hourly(var_name)

fitted_medians(var_name, doys=None)

Medians of the fitted seasonal distribution.

Usefull to construct `climate_signal`s

Parameters:

var_namestr or sequence of strings

doys1d array

If None, sim_times is used. If that is not available, data_doys.

property output_resolution

In hours.

property primary_var_ii

The row index of the primary variable.

property sim_doys

Days of year of aggregated output data.

property start_hour_of_src

property sum_interval_dict

Maps var_name to sum_interval.

varwg.core.base.VGBase

alias of Base

varwg.core.base.daily_wind(met, sum_interval=24)

Aggregates daily wind direction and speed from hourly values.

varwg.core.base.detrend(values)

Detrend by subtracting the linear regression function.

varwg.core.base.dyresm2array(filepath, *args, **kwds)

varwg.core.base.met2array(filepath, met_kwds=None, as_array_kwds=None, sumup_kwds=None)

varwg.core.base.met_as_array(met, p=0, T=None, var_names=None)

Returns the met - dictionary as an array. Rows are sorted alpha-numerically according to the variable names.

varwg.core.base.read_met(filepath=None, minimum_water_temp=0.1, delimiter='\t', verbose=True, main_diff=None, with_conversions=False, **kwds)

varwg.core.core module

class varwg.core.core.VarWG(var_names, met_file=None, sum_interval=24, max_nans=12, plot=False, separator='\t', refit=None, detrend_vars=None, verbose=True, data_dir=None, cache_dir=None, dump_data=True, non_rain=None, rain_method='regression', conf_update=None, station_name=None, infill=False, fit_kwds=None, **met_kwds)

Bases: Plotting

A Vector-Autoregressive weather generator.

>>> my_vg = VG(("theta", "Qsw", "ILWR", "rh", "u", "v"))
>>> my_vg.fit()
>>> times_out, sim_data = my_vg.simulate()

Attributes:

K

The number of variables.

T_data

Length of the input data time series (‘hourly’).

T_summed

Length of aggregated input data (‘daily’).

cov_trans

The covariance matrix of the transformed data.

data_doys

Days of year of aggregated input data.

data_doys_raw

Days of year of raw (hourly) input data.

dis_doys

Days of year of disaggregated output data.

output_resolution

In hours.

primary_var_ii

The row index of the primary variable.

sim_doys

Days of year of aggregated output data.

start_hour_of_src

sum_interval_dict

Maps var_name to sum_interval.

ylabels

Methods

disaggregate([var_names_dis, event_dt, ...])	Disaggregate variables to hourly time steps by drawing from the residuals of the measured data
fit([p, q, p_max, seasonal, ex, ex_kwds, ...])	p is the order of autoregressive process.
fit_seasonal([refit, values, doys])	This is here, so that when initialized directly, seasonal distributions are fitted, but the existing fit to the data defined in the config file is not overwritten.
fitted_medians(var_name[, doys])	Medians of the fitted seasonal distribution.
plot_VAR_par(**kwds)	Plots the parameters of the VAR-Process.
plot_all(*args, **kwds)	Executes every method of Plotting that starts with "plot_".
plot_autocorr([maxlag, title, var_names])	Plots autocorrelation of transformed and simulated data (if the latter is available).
plot_cross_corr([max_lags, figsize])	Plot cross correlations.
plot_daily_cycles([var_names, rain_thresh])	Plot cycle of hourly means over the year.
plot_daily_fit([var_names])	Plots daily scatter with fitted densities and a histogram of the transformed quantiles.
plot_doy_scatter([var_names, opacity, hourly])	Plot variables over doys.
plot_episode_hists([var_names])	Plots histograms of episode duration and amplitude grouped by variable.
plot_hourly_fit([var_names])	Plots hourly scatter with fitted densities and a histogram of the transformed quantiles.
plot_meteogram_hourly([var_names, figs, ...])	All variables over time in subplots.
plot_monthly_hists([var_names, bins, figs, axss])	Plot histograms grouped by month and variable.
plot_monthly_hists_hourly([var_names, bins])	Plot histograms grouped by month and variable.
plot_psd([var_names, hourly])	Plots power spectral density using matplotlib.pyplot.psd.
plot_scaling([agg_funcs, max_agg_len, normalize])	Plot mean/std/skew over aggregation length.
predict([dtimes, data_past, T, n_realizations])	Predict the next T steps given the past data.
random_dryness([T, start_str, stop_str, ...])	This is a very special method designed to generate dry hot events for Lake Kinneret.
simulate([T, mean_arrival, disturbance_std, ...])	Simulate based on data from __init__ and the fit from calling self.fit().
to_dyresm(outfilepath[, ts, info, wind_fct, ...])	Converts output to a DYRESM input file.
to_glm(outfilepath)	Converts output to a GLM input file.
to_gotm(outfilepath[, sw_outfilepath, ...])	Converts output to a GOTM input file set (met and separate short-wave file).

dist_sol_hourly
infill_trans_nans
plot_candidates
plot_corr
plot_diff_spaces
plot_doy_scatter_residuals
plot_exceedance_daily
plot_exeedance_hourly
plot_hyd_year_sums
plot_meteogram_daily
plot_meteogram_trans
plot_prediction
plot_qq
plot_seasonal_corrs
plot_spaces
plot_windrose
print_means
to_df

property cov_trans

The covariance matrix of the transformed data. If seasonal is True, this is of (365, K, K)-shape.

disaggregate(var_names_dis=None, event_dt=None, factors=None, doy_tolerance=15, latitude=None, longitude=None)

Disaggregate variables to hourly time steps by drawing from the residuals of the measured data

Parameters:

var_names_dislist of str, optional

names of variables to be disaggregated. if None all variables are disaggregated

event_dtarray of datetimes, optional

days on which factors will be applied to change daily cycle amplitudes.

factorssequence of length K

see event_dt

Returns:

times_outarray of datetime objects

sim_sea_disarray

simulated variables in hourly time steps. the last day gets lost

fit(p=None, q=0, p_max=10, seasonal=True, ex=None, ex_kwds=None, extro=False, doy_width=60, fft_order=2, *args, **kwds)

p is the order of autoregressive process. q the order of the moving average process If p is not given, it is selected using the Schwartz Information Criterion.

infill_trans_nans()

plot_prediction(dtimes=None, data_past=None, T=1, hindcast=True)

predict(dtimes=None, data_past=None, T=1, n_realizations=1)

Predict the next T steps given the past data.

Parameters:

dtimesNone or (p,) ndarray, optional

datetimes of past timesteps, if None, the last p datetimes from the input data are used

data_pastNone or (K, p)) ndarray, optional

Past on which to predict the future. p is the order of the VAR - process (it is stored in self.p after self.fit() is called). If None, data from the input is used.

Tint, optional

number of timesteps to predict.

n_realizationsint, optional

number of realizations of prediction

Returns:

prediction(K, T) or (K, T, n_realizations) ndarray

print_means()

random_dryness(T=None, start_str=None, stop_str=None, n_events=None, duration_min=2, duration_max=7, month_start=6, month_end=9, event_dryness=0.4)

This is a very special method designed to generate dry hot events for Lake Kinneret.

Parameters:

TNone or int, optional

Number of timesteps to simulate. T=None produces as many simulation timesteps as there were in the data used for fitting the VARMA process.

start_strstr of format ‘%m-%d %H’, optional

String representation of the start date of the simulation.

stop_strstr of format ‘%m-%d %H’, optional

String representation of the end date of the simulation.

n_eventsint or None

Number of events to generate. If none there will be an average of 3 events per year.

duration_minint

Minimum length of events

duration_maxint

Maximum length of events

month_startint

Month when the events can start to happen (january = 1)

month_endint

Latest month in which the events can happen (january = 1)

event_drynessfloat

The value of relative humidity that will be set during the events.

Returns:

rh_signalndarray, dtype=float

To be used as climate_signal in VG.simulate

event_dtndarray, dtype=object (datetime)

Datetimes of the events. Can be passed to VG.disaggregate later.

simulate(T=None, *, mean_arrival=None, disturbance_std=None, theta_incr=None, r_fact=None, theta_grad=None, fixed_variables=None, primary_var='theta', scale_prim=True, climate_signal=None, start_str=None, stop_str=None, random_state=None, ex=None, ex_kwds=None, seed_before_sim=None, resample=False, res_kwds=None, asy=False, residuals=None, sim_func=None, sim_func_kwds=None, phase_randomize=False, phase_randomize_vary_mean=True, return_rphases=False, rphases=None, p_kwds=None, taboo_period_min=None, taboo_period_max=None, conversions=None)

Simulate based on data from __init__ and the fit from calling self.fit(). To choose a specific order of AR and MA call fit(p, q) before calling simulate. Otherwise fit will be called without parameters, so p and q will be guessed by applying an information criterion.

Parameters:

TNone or int, optional

Number of timesteps to simulate. T=None produces as many simulation timesteps as there were in the data used for fitting the VARMA process.

mean_arrivalfloat, optional

average length of warm/cold periods [d]

disturbance_stdfloat, optional

standard deviation of deviation of warm/cold periods from mean

theta_incrfloat, optional

increase in average of temperature [deg C]

r_factfloat, optional

factor to increase precipitation

theta_gradfloat, optional

gradient increase in average of temperature [deg C / (T)]

fixed_variablesdictionary mapping var_names to None or (T,) ndarrays

Keeps the provided time - series fixed. Mappings to None indicate that the input data should be used as a fixed variable. Can be used to simulate hierarchically.

primary_varstr, one of var_names from __init__ or sequence of those,

optional

All disturbances (mean_arrival, distrubance_std, theta_incr, theta_grad and climate_signal) correspond to changes in this variable.

scale_primboolean, optional

Allow primary variables to influence each other. Only effective when using multiple primary variables!

climate_signal(T,) ndarray

A time series of the ‘primary_var’. Differences between ‘climate_signal’ and the seasonal mean are used to perturb the VAR process.

start_strstr of format ‘%m-%d %H’, optional

String representation of the start date of the simulation.

stop_strstr of format ‘%m-%d %H’, optional

String representation of the end date of the simulation.

random_statestr, tuple as returned by numpy.random.get_state() or

None, optional

Sets the random state before simulating. str is interpreted as a pickle file containing the random state. tuple is interpreted as the random state itself.

ex(T,) ndarray or function

External variable. If given as a function, ex_t will be generated by calling ex(Y[:t], **ex_kwds), with Y being the simulated values.

ex_kwdsNone or dict

Keyword arguments to be passed to ex.

seed_before_simNone or int, optional

Seed the Random Generator before simulating. The integer passed will be used as seed (also if it is a 0). (_scenario_parameters also draws random numbers).

loc_shiftbool, optional

Change on the primary variables will be removed after simulation and added to the backtransformed variables.

res_kwdsdict or None, optional

Keywords for the resampler, if not None, use resampling. See time_series_analysis.resample.resample for allowed parameters.

residuals(K, T+) ndarray

Residuals with which to run the time series model. If more than T columns are supplied the additional time steps (at the beginning) will be used for spin-up.

sim_funcNone or callable, optional

Callback to replace the time series model.

sim_func_kwdsNone or dict

Extra keyword arguments for sim_func

conversionsiterable of callables

Returns:

sim_times(T,) ndarray of datetime objects

sim_sea(K, T) float ndarray

to_df(kind='hourly input', var_names=None, *, with_conversions=True)

to_dyresm(outfilepath, ts=3600, info=None, wind_fct=1.0, output_rh=False)

Converts output to a DYRESM input file.

Parameters:

outfilepathstr

dyresm meteorological bc file

tsint

timestep in dyresm-met-file in seconds, should be divisor of 86400

infoinformation text string in file header

wind_fctfloat

factor for wind speed. Default: 1.0 (for Konstanz wind data: 1.3)

to_glm(outfilepath)

Converts output to a GLM input file.

Parameters:

outfilepathpath/filename

glm meteorological bc file

to_gotm(outfilepath, sw_outfilepath=None, rain_outfilepath=None)

Converts output to a GOTM input file set (met and separate short-wave file).

Parameters:

outfilepathpath/filename

GOTM meteorological bc file

sw_outfilepathpath/filename

GOTM short-wave bc file

rain_outfilepathpath/filename

GOTM precipitation bc file

varwg.core.core.delete_cache()

varwg.core.core.dump_data(times_, data, var_names, p, q, extra='', random_state=None, out_dir=None, conversions=None)

Dumps the given datetimes and data in an ascii file and random state in a pickle file.

varwg.core.core.interannual_variability(T, mean_arrival=30, disturbance_std=0.1)

Generate a time series used as a disturbance within the std-norm world. Disturbances are simulated as gaussian changes occuring after t timesteps, where t is drawn from an exponential distribution.

varwg.core.core.metfile2df(met_file)

varwg.core.core.metfile2hdf5(met_file, h5_filename=None, key='met')

varwg.core.core.outfile2df(outfile)

varwg.core.core.seasonal_back(dist_sol, norm_data, var_names, doys=None, solution_template='%s', pass_doys=True, var_names_back=None, mean_shifts=None)

Transform variables from normal to real-world marginals.

Parameters:

dist_solmapping like dict or shelve object

Distribution fitting parameters.

norm_data(K, T) array

Normally distributed data.

var_names(K,) sequence of str

Variable names used as dist_sol[solution_template % var_name].

doysNone or (T,) float array, optional

Only needed for different doys than the calibration dataset.

solution_templatestr, optional

Used in the form dist_sol[solution_template % var_name]

Returns:

data(K, T) array

Re-transformed data

varwg.core.core.sim_byvariable(values, times, p=2, aggregation='%w')

varwg.core.core.sw_diurnal(date, daily_sw_data, del_t=3600)

add standard daily cycle to daily mean sw data Parameters ———- date: array

date in unix timestamps

daily_sw_data: array del_t: int

timestep of output data in seconds, default = 3600s

Returns:

date_: array

output date in unix timestamps

data_: array

sw data in del_t-steps

varwg.core.plotting module

class varwg.core.plotting.Plotting(var_names, met_file=None, sum_interval=24, max_nans=12, plot=False, separator='\t', refit=None, detrend_vars=None, verbose=False, data_dir=None, cache_dir=None, seasonal_fitting=False, dump_data=False, non_rain=None, rain_method=None, fit_kwds=None, **met_kwds)

Bases: Base

To be used indirectly (by the VG class) or directly to plot meteorological data (met_files, pandas dataframe).

Caching features will be disabled to not overwrite anything more useful.

Attributes:

K

The number of variables.

T_data

Length of the input data time series (‘hourly’).

T_summed

Length of aggregated input data (‘daily’).

data_doys

Days of year of aggregated input data.

data_doys_raw

Days of year of raw (hourly) input data.

dis_doys

Days of year of disaggregated output data.

output_resolution

In hours.

primary_var_ii

The row index of the primary variable.

sim_doys

Days of year of aggregated output data.

start_hour_of_src

sum_interval_dict

Maps var_name to sum_interval.

ylabels

Methods

fit_seasonal([refit, values, doys])	This is here, so that when initialized directly, seasonal distributions are fitted, but the existing fit to the data defined in the config file is not overwritten.
fitted_medians(var_name[, doys])	Medians of the fitted seasonal distribution.
plot_VAR_par(**kwds)	Plots the parameters of the VAR-Process.
plot_all(*args, **kwds)	Executes every method of Plotting that starts with "plot_".
plot_autocorr([maxlag, title, var_names])	Plots autocorrelation of transformed and simulated data (if the latter is available).
plot_cross_corr([max_lags, figsize])	Plot cross correlations.
plot_daily_cycles([var_names, rain_thresh])	Plot cycle of hourly means over the year.
plot_daily_fit([var_names])	Plots daily scatter with fitted densities and a histogram of the transformed quantiles.
plot_doy_scatter([var_names, opacity, hourly])	Plot variables over doys.
plot_episode_hists([var_names])	Plots histograms of episode duration and amplitude grouped by variable.
plot_hourly_fit([var_names])	Plots hourly scatter with fitted densities and a histogram of the transformed quantiles.
plot_meteogram_hourly([var_names, figs, ...])	All variables over time in subplots.
plot_monthly_hists([var_names, bins, figs, axss])	Plot histograms grouped by month and variable.
plot_monthly_hists_hourly([var_names, bins])	Plot histograms grouped by month and variable.
plot_psd([var_names, hourly])	Plots power spectral density using matplotlib.pyplot.psd.
plot_scaling([agg_funcs, max_agg_len, normalize])	Plot mean/std/skew over aggregation length.

dist_sol_hourly
plot_candidates
plot_corr
plot_diff_spaces
plot_doy_scatter_residuals
plot_exceedance_daily
plot_exeedance_hourly
plot_hyd_year_sums
plot_meteogram_daily
plot_meteogram_trans
plot_qq
plot_seasonal_corrs
plot_spaces
plot_windrose

fit_seasonal(refit=None, values=None, doys=None)

This is here, so that when initialized directly, seasonal distributions are fitted, but the existing fit to the data defined in the config file is not overwritten.

plot_VAR_par(**kwds)

Plots the parameters of the VAR-Process.

plot_all(*args, **kwds)

Executes every method of Plotting that starts with “plot_”. There are quite a number of those. You have been warned.

plot_autocorr(maxlag=7, title=None, var_names=None, *args, **kwds)

Plots autocorrelation of transformed and simulated data (if the latter is available).

Parameters:

maxlagint, optional

Maximum number of lags to plot

plot_candidates(figs=None, axss=None)

plot_corr(ygreek=None, hourly=False, trans=False, *args, **kwds)

plot_cross_corr(max_lags=7, figsize=None)

Plot cross correlations.

plot_daily_cycles(var_names=None, rain_thresh=None, **fig_kw)

Plot cycle of hourly means over the year.

Climbing Mt. Allon.

Parameters:

var_namessequence of str, optional

Which variables to plot. None means plot all.

rain_threshNone, True or float, optional

Which elemts to include based on rain threshold. If True, it is inferred from the configuration file (dist_kwds[“R”][“threshold”])

plot_daily_fit(var_names=None, *args, **kwds)

Plots daily scatter with fitted densities and a histogram of the transformed quantiles.

Parameters:

var_namessequence of str, optional

Which variables to plot. None means plot all.

plot_quantilesboolean, optional

Plot histograms of the quantiles?

plot_fourierboolean, optional

Plot the fourier fit to the seasonally changing distribution parameters.

plot_monthlyboolean, optional

Plot histograms showing how good the fit is on monthly separated data.

opacityfloat, optional

Opacity of scatter.

n_binsint or sequence, optional

Passed on to np.histogram when plotting histograms of quantiles.

kdeboolean, optional

Plot a kernel density estimation on the histograms of quantiles.

s_kwdsNone or dict, optional

Keyword arguments passed on to plt.scatter.

plot_diff_spaces(var_names=None, diff=1, opacity=0.4)

plot_doy_scatter(var_names=None, opacity=0.4, hourly=False, **f_kwds)

Plot variables over doys. Measurement and backtransformed simulated data will be plotted, if it is available.

Parameters:

var_namessequence of str, optional

Which variables to plot. None means plot all.

opacityfloat in the range (0,1], optional

Opacity of the scatter points.

.. plot::

met_vg = VG((“theta”,)) sim_times, sim = met_vg.simulate() met_vg.plot_doy_scatter()

plot_doy_scatter_residuals(var_names=None, opacity=0.4)

plot_episode_hists(var_names=None)

Plots histograms of episode duration and amplitude grouped by variable.

plot_exceedance_daily(thresh=None, fig=None, axs=None, *args, **kwds)

plot_exeedance_hourly(thresh=None, fig=None, axs=None, *args, **kwds)

plot_hourly_fit(var_names=None, *args, **kwds)

Plots hourly scatter with fitted densities and a histogram of the transformed quantiles.

Parameters:

var_namessequence of str, optional

Which variables to plot. None means plot all.

plot_quantilesboolean, optional

Plot histograms of the quantiles?

plot_fourierboolean, optional

Plot the fourier fit to the seasonally changing distribution parameters.

plot_monthlyboolean, optional

Plot histograms showing how good the fit is on monthly separated data.

opacityfloat, optional

Opacity of scatter.

n_binsint or sequence, optional

Passed on to np.histogram when plotting histograms of quantiles.

kdeboolean, optional

Plot a kernel density estimation on the histograms of quantiles.

s_kwdsNone or dict, optional

Keyword arguments passed on to plt.scatter.

plot_hyd_year_sums(fig=None, ax=None, alpha=0.5, *args, **kwds)

plot_meteogram_daily(var_names=None, figs=None, axss=None, figsize=None, plot_sim_sea=True, obs=None, sim=None, plot_daily_bounds=True, p_kwds=None, h_kwds=None, **f_kwds)

plot_meteogram_hourly(var_names=None, figs=None, axss=None, plot_sim_sea=True, obs=None, sim=None, combine=False, **f_kwds)

All variables over time in subplots. (Not as nice as the meteogram from varwg.meteo)

plot_meteogram_trans(var_names=None, figs=None, axss=None, **f_kwds)

plot_monthly_hists(var_names=None, bins=20, figs=None, axss=None)

Plot histograms grouped by month and variable.

plot_monthly_hists_hourly(var_names=None, bins=20)

Plot histograms grouped by month and variable.

plot_psd(var_names=None, hourly=False, *args, **kwds)

Plots power spectral density using matplotlib.pyplot.psd.

Parameters:

var_namessequence of str, optional

Which variables to plot. None means plot all.

hourlyboolean, optional

Plot hourly and disaggregated values.

*args, **kwds

Passed on to matplotlib.pyplot.psd

plot_qq(var_names=None, figsize=None, lines=True, trans=False, color='gray', fig=None, axs=None, obs=None, *args, **kwds)

plot_scaling(agg_funcs=(<function mean>, <function std>, <function skew>, <function kurtosis>), max_agg_len=730, normalize=True, **fig_kw)

Plot mean/std/skew over aggregation length.

plot_seasonal_corrs(var_names=None, trans=False, figsize=None, fig=None, axs=None)

plot_spaces(var_names=None, opacity=0.4)

plot_windrose(figsize=None, seasonal=False, *args, **kwds)

property ylabels

varwg.core.plotting.VGPlotting

alias of Plotting

varwg.core.plotting.append_fa(fig_ax, figs=None, axs=None)

varwg.core.plotting.var_names_greek(var_names)

Module contents