MRC-Feed

This forum is dedicated to packaging and translating :

On those activities rely the making of Mageia Linux Distribution.

Post all questions and information about packaging and translating : feedbacks, discussion about guidelines, packaging practices...

MRC-Feed

Postby xxblx » May 12th, '14, 12:30

Hello.

MRC-Feed is a small tool written in Lua 5.1. It generate list of RPM packages in repository and add new entries about recently added into repository packages to RSS-Feed.

How it work: Lua script generate HTML files with tables which contain full list of repository rpm packages and add new <item>s about recently added packages to XML files of RSS feeds.

MRC-Feed is not designed for big repositories! Because it doesn't use database management system (like MySQL, MariaDB, SQLite, etc...). All information (repository packages list and packages details) store in files pkgs_32.lua, pkgs_64.lua, pkgs_noarch.lua in pkgs_db/mageia<release_number> directory. For example, information about packages for Mageia 4 i586 will stored in pkgs_db/mageia4/pkgs_32.lua.
Open spoiler to see details about pkgs_32.lua/pkgs_64.lua/pkgs_noarch.lua structure:
Spoiler:
pkgs_32.lua, pkgs_64.lua, pkgs_noarch.lua are contain hash-tables (key = value) where keys are rpms (in fact, keys list is rpms list) and values are another hash-table with package details.

It's look like this:
Code: Select all
-- hash-table with packages for Mageia i586
pkgs_lst_32 = {}

pkgs_lst_32["wxlua-2.8.12.3-4.mga4.mrc.i586.rpm"] = {
    name = "wxlua",
    version = "2.8.12.3",
    group = "Development/Other",
    packager = "xxblx",
    summary = "wxLua bindings for Lua 5.2 and wxWidgets 2.9.5",
}


MRC-Feed used for generating rpm packages list of Mageia Russian Community repositories. You can check the result (MRC-Feed output) here - html-files with rpms lists: link1 and link2; xml-files of rss-feeds: for mga2, for mga3, for mga4.

MRC-Feed source available on bitbucket.org (License: GNU GPL v3). See details (how to use, dependencies, etc...) in README file.
Feel free to get and use it. May be it will be helpful for small unofficial repositories.

Thanks.
Oleg.
Mageia x86_64, E17, PekWM
Oleg (xxblx) | Mageia Russian Community
User avatar
xxblx
 
Posts: 70
Joined: May 23rd, '12, 21:32
Location: Russia, Vologda

Re: MRC-Feed

Postby doktor5000 » May 17th, '14, 01:18

Why not use e.g. http://mageia.madb.org/ or try to get your repositories listed there?
Cauldron is not for the faint of heart!
Caution: Hot, bubbling magic inside. May explode or cook your kittens!
----
Disclaimer: Beware of allergic reactions in answer to unconstructive complaint-type posts
User avatar
doktor5000
 
Posts: 17630
Joined: Jun 4th, '11, 10:10
Location: Leipzig, Germany

Re: MRC-Feed

Postby xxblx » May 17th, '14, 14:41

Mageia App Db is a great and powerful web tool for packages management. It's a complex and large web application, but we (for mrc-repo) just need to generate packages list and notify users by RSS. So, I think that it's makes no sense to use a big app when you need just a couple of its functions. That was the reason for MRC-Feed creation.
MRC-Feed is a small script. It doesn't need database management system, it doesn't make high CPU load and doesn't need a lot of RAM. We added MRC-Feed to crontab and it execs once a day. Output of MRC-Feed are html-files with actual packages list in repo and new <item>s in xml-files of RSS-feeds.
Mageia x86_64, E17, PekWM
Oleg (xxblx) | Mageia Russian Community
User avatar
xxblx
 
Posts: 70
Joined: May 23rd, '12, 21:32
Location: Russia, Vologda

Re: MRC-Feed

Postby doktor5000 » May 17th, '14, 18:52

Sounds reasonable for your use case - and thanks for the explanation :)
Cauldron is not for the faint of heart!
Caution: Hot, bubbling magic inside. May explode or cook your kittens!
----
Disclaimer: Beware of allergic reactions in answer to unconstructive complaint-type posts
User avatar
doktor5000
 
Posts: 17630
Joined: Jun 4th, '11, 10:10
Location: Leipzig, Germany

Re: MRC-Feed

Postby xxblx » Jul 14th, '14, 18:10

Hello.
I working under re-write MRC-Feed in Python. I have few reasons for it.
First of all, Mageia's RPM doesn't have Lua support. So, Lua can't to get direct access to rpm and need to use shell for it (os.execute or os.popen). That do code ugly and dirty. It's really bad.
RPM have Python support. Python-rpm module allow to get access to rpm from python code easy and fast. It's simple and clear.
Spoiler:
Code: Select all
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import rpm
import sys

try:
   pkg = sys.argv[1]
except:
   pkg = "python-rpm"

ts = rpm.TransactionSet()
res = ts.dbMatch("name", pkg)
for i in res:
  print "%s-%s-%s" % (i['name'], i['version'], i['release'])
  print "Group: ", i["group"]
  print "Packager", i["packager"]
 
  print "\nFiles in package:"
  for v in i.fiFromHeader():
     print(v[0])

Code: Select all
$ python ./script.py
python-rpm-4.11.1-8.mga4
Group:  Development/Python
Packager tv <tv>

Files in package:
/usr/lib/python2.7/site-packages/rpm
/usr/lib/python2.7/site-packages/rpm/__init__.py
/usr/lib/python2.7/site-packages/rpm/__init__.pyc
/usr/lib/python2.7/site-packages/rpm/__init__.pyo
/usr/lib/python2.7/site-packages/rpm/_rpmbmodule.so
/usr/lib/python2.7/site-packages/rpm/_rpmmodule.so
/usr/lib/python2.7/site-packages/rpm/_rpmsmodule.so
/usr/lib/python2.7/site-packages/rpm/transaction.py
/usr/lib/python2.7/site-packages/rpm/transaction.pyc
/usr/lib/python2.7/site-packages/rpm/transaction.pyo

Planned features for new version: support of http and ftp repositories (lua-version support only local repositories); using sqlite3 as database management system for data storage.

Web-page feed.mageialinux.ru already turned off. Old MRC-Feed git-repository on bitbucket will be closed soon (temporary). After new version release old version will be moved to another branch or to a new repo.
Mageia x86_64, E17, PekWM
Oleg (xxblx) | Mageia Russian Community
User avatar
xxblx
 
Posts: 70
Joined: May 23rd, '12, 21:32
Location: Russia, Vologda

Re: MRC-Feed

Postby doktor5000 » Jul 14th, '14, 20:02

xxblx wrote:First of all, Mageia's RPM doesn't have Lua support.

How did you test?
Code: Select all
[doktor5000@Mageia4 ~]$ rpm --showrc | grep -i lua
    rpmlib(BuiltinLuaScripts) = 4.2.2-1
-14: _changelog_trimtime        %{lua:print(os.time() - 3 * 365 * 86400)}
-14: apply_patches      %{lua: keys = {}; for i, p in ipairs(patches) do print(rpm.expand("%{_patch} -s -p1 -b --suffix " .. string.format(".%04d", patches_num[p]) .. " --fuzz=%{_default_patch_fuzz} -i " .. p .. "\n")) end }
%{lua:
-14: patches    %{lua: for i, p in ipairs(patches) do print(p.." ") end}
-14: sources    %{lua: for i, s in ipairs(sources) do print(s.." ") end}

Lua support is present since ages from what I remember.

Related, the lua packaging itself has changed recently in cauldron: https://ml.mageia.org/wwsympa-wrapper.f ... 00552.html
Cauldron is not for the faint of heart!
Caution: Hot, bubbling magic inside. May explode or cook your kittens!
----
Disclaimer: Beware of allergic reactions in answer to unconstructive complaint-type posts
User avatar
doktor5000
 
Posts: 17630
Joined: Jun 4th, '11, 10:10
Location: Leipzig, Germany

Re: MRC-Feed

Postby xxblx » Jul 14th, '14, 22:20

Lua support is present since ages from what I remember.

Yes. It's my mistake. I was incorrect. Of course, RPM have Lua support. AFAIK, rpm have embedded lua interpreter. It allow to do, for example
Code: Select all
%pre -p <lua>
print("Yep! You can use Lua =)")


In my previous message I mean lua rpm bindings, external rpm module (like python-rpm) for Lua which will allow to get direct access from lua code to package management (get rpm's info, search, install/uninstall, etc...).
I mean something like this
Code: Select all
-- it's a fake code just for example, lua doesn't have rpm module like python-rpm (as I know)
local rpm = require("rpm")

local m = dbMatch("name", "python-rpm")

for i, v in pairs(m) do:
  print(i.." - "..v)
end

It's not possible to do from Lua. So, it's need to use shell:
Code: Select all
local rpm_result = io.popen("rpm -qpi pkg.rpm"):read("*a")

local pkg = {}

pkg.name = string.match(rpm_result, "Name%s*:%s*([%a%d%p].-)\n")
pkg.version = string.match(rpm_result, "Version%s*:%s*([%a%d%p].-)\n")
pkg.group = string.match(rpm_result, "Group%s*:%s*([%a%d%p].-)\n")
pkg.packager = string.match(rpm_result, "Packager%s*:%s*([%a%d%p].-)\n")
pkg.summary = string.match(rpm_result, "Summary%s*:%s*([%a%d%p].-)\n")

print("pkg info:")
for i, v in pairs(pkg):
  print(i.." - "..v)
end

It's not good.

Easier to use Python. I already posted snippet in previous post - how to easy get package's info in python (under spoiler).

p.s. it's not a python PR :D I much more like lua, but lua here just will create additional difficulties in future. easier to re-write script now, while it's a small.
Mageia x86_64, E17, PekWM
Oleg (xxblx) | Mageia Russian Community
User avatar
xxblx
 
Posts: 70
Joined: May 23rd, '12, 21:32
Location: Russia, Vologda


Return to Packaging and translating for Mageia

Who is online

Users browsing this forum: No registered users and 1 guest

cron